Open demands for corpus analysis tools - a user-centered study
Der Philosophischen Fakultät
der Friedrich-Alexander-Universität Erlangen-Nürnberg
zur
Erlangung des Doktorgrades Dr. phil.
vorgelegt vonVerena Ruth Hildegard Lyding

Als Dissertation genehmigt
von der Philosophischen Fakultät
der Friedrich-Alexander-Universität Erlangen-Nürnberg
Tag der mündlichen Prüfung: 17.12.2021

Vorsitzende/r des Promotionsorgans: Prof. Dr. Thomas Demmelhuber

Gutachter/in: Prof. Dr. Stephanie Evert
Prof. Dr. Ulrich Heid

3
Abstract

It was not before the advent of powerful computers that corpus linguistics hasdeveloped into a widely applied research methodology. Indeed, corpus linguistics heavily relies on computer-powered analysis tools. They get used ona daily basis by corpus linguists to retrieve examples and analyze authentic
data from corpora of extensive sizes. Despite their indisputable importance,
repetitive remarks highlight the fact that corpus analysis tools have evolved
little since their early days. Concordances, frequency lists and collocation
extraction still constitute the core functionalities of most corpus tools.
With the aim to incentivize new functional developments, this thesispresents research on open demands in current corpus research practice andrelated requirements for tools support. It builds on the assumption that moreuser-centered research is needed to bridge the gap between mainly computationally trained tool developers and their linguistic expert users, who comewith specialized domain knowledge and often sophisticated analytical needs.
The research is approached by means of three user investigations that enquire
about corpus research workows and analysis activities as well as theoreticalprinciples and methodological considerations in corpus linguistics research
practice. This way a comprehensive picture of the corpus usage situation
is assembled by combining insights from open ended enquiries (interviews)
with quantitative results on selected aspects of the corpus analysis scenario(questionnaire) derived from enquiries with overall more than 100 corpus
users. Based on the results, a range of open demands for corpus researchand tools are identied and discussed. They relate to (1) corpus resources,(2) general aspects of tools, (3) corpus analysis procedures, and (4) bestpractices. The results show that open demands address challenges on very
dierent operational levels, ranging from the availability of corpus resources
and reliable annotations, technical requirements related to scalability and interoperability issues, usability and technical and methodological skills up toproper functional demands. The thesis discusses potential paths to address
the open demands, and provides pointers to recent developments in corpus
linguistics and related elds, in particular computational linguistics and natural language processing as well as linguistic information visualization.
The research contribution of this thesis is twofold. On the methodological level, it elaborates on methods and challenges for user-centered researchon tools for open-ended tasks and provides entrance points for further usercentered research by identifying and organizing, as reference, the basic building blocks of corpus linguistics research. On the content level, it provides
rst insights on user perspectives and needs related to the corpus research
practice. It describes concrete demands and discusses paths to their solution. This way, it prepares the ground for further in-depths studies and usercentered developments of new corpus functionalities for specic demands.

4
Abstract (German version)

Oene Anforderungen an Korpusanalysewerkzeuge - eine nutzerzentrierte Studie
Mit der Verbreitung leistungsstarker Computer begann die Korpuslinguistik als Forschungsmethode erstmals weitreichende Anwendung zu nden.
Tatsächlich bedient sie sich in groÿem Ausmaÿ computergestützter Analysewerkzeuge, welche tagtäglich zum Einsatz kommen, um Korpora von beachtlicher Gröÿe zu analysieren. Trotz ihrer zentralen Position wird wiederholt
herausgestellt, dass sich auf Funktionenebene seit Anbeginn wenig Neuerungen verzeichnen lassen. Konkordanzen, Frequenzlisten und Kollokationsextraktion bilden bis heute die Kernfunktionen der meisten Korpuswerkzeuge.
Mit dem Ziel neue Entwicklungen anzustoÿen, präsentiert diese Arbeit
Forschung zu oenen Anforderung in der Korpusforschungspraxis. Unter
der Annahme, dass mehr nutzerorientierte Studien erforderlich sind, um
die technisch orientierten Entwicklungen von Korpuswerkzeugen mit den
oft anspruchsvollen und fachbezogenen Analysebedürfnissen von KorpuslinguistInnen in Einklang zu bringen, wurde die Forschung anhand von drei
Nutzerbefragungen umgesetzt, welche Informationen zu Korpusanalyseworkows und -aktivitäten, sowie zu theoretischen und methodologischen Prinzipien sammeln. Indem Einsichten aus oenen Befragungen (Interviews) mit
quantitativen Ergebnissen zu ausgewählten Aspekten des Korpusanalyseszenarios (Fragebogen) aus Befragungen mit insgesamt über 100 TeilnehmerInnen kombiniert wurden, konnte ein umfassendes Bild der Korpusnutzersituation erarbeitet werden. Es mündet in die Beschreibung einer Reihe offener Anforderungen für die korpusbasierte Forschung und Unterstützung
durch Werkzeuge, welche bezogen auf vier Aspekte diskutiert werden: (1)Korpusressourcen, (2) allgemeine Aspekte von Werkzeugen, (3) Korpusanalyseprozeduren) und (4) Best Practices. Die Ergebnisse zeigen, dass oene
Anforderungen sich auf Herausforderungen auf unterschiedlichen operativen
Ebenen beziehen. Diese reichen von der Verfügbarkeit von Korpusressourcen
und verlässlichen Annotationen, über technische Anforderungen hinsichtlich
Skalierbarkeit, Interoperabilität und Nutzerfreundlichkeit, bis hin zu Nutzerskills und schlieÿlich funktionellen Aspekten. Diese Thesis diskutiert potentielle Antworten und Lösungswege und stellt Bezüge zu jüngeren Entwicklungen in der Korpuslinguistik und benachbarten Disziplinen her, insbesondere
zu denen der Computerlinguistik, des Natural Language Processing und derVisualisierung linguistischer Daten.
Die Arbeit leistet einen zweifachen Forschungsbeitrag. Auf methodischer
Ebene erörtert die Arbeit Herausforderungen und Ansätze für die nutzerzentrierte Forschung zu Werkzeugen für oene Aufgabenstellungen. Auch bietet
die Arbeit Einstiegspunkte für weitere nutzerzentrierte Forschung, indem die5Grundpfeiler der korpuslinguistischen Forschung identiziert und systematisch beschrieben werden und somit ein Referenzrahmen, der bisher fehlte,
hergestellt wird. Auf inhaltlicher Ebene liefert diese Forschungsarbeit erste
Einsichten zu Nutzerperspektiven in Bezug auf die tägliche Praxis der Korpusforschung und leitet daraus eine Reihe von Anforderungen ab. Die Anforderungen werden nach inhaltlichen Gruppen strukturiert beschrieben und
Wege zu deren Lösung werden diskutiert. Damit bereitet diese Arbeit den
Weg für weitere vertiefende Studien zu Nutzeranforderungen und ermutigt
nutzerzentrierte Entwicklungen neuer Korpusfunktionalitäten in Bezug auf
die diskutierten Anforderungen.

6Contents

1 Introduction
1.1 Status quo of user-centered research in corpus linguistics . . .
1.2 Research objective and approach . . . . . . . . . . . . . . . .
1.3 Scope and timeliness . . . . . . . . . . . . . . . . . . . . . . .
1.4 Structure of the thesis . . . . . . . . . . . . . . . . . . . . . .

9
12
13
15
16

2 Corpus Linguistics
2.1 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Objectives and theoretical principles . . . . . . . . . . . . . .
2.3 Linguistic application areas . . . . . . . . . . . . . . . . . . .
2.4 Research methodology . . . . . . . . . . . . . . . . . . . . . .
2.5 Data and tools . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6 User and usability research in CL . . . . . . . . . . . . . . . .
2.7 Conclusions and aims of this thesis . . . . . . . . . . . . . . .

19
21
23
29
32
37
49
54

3 User-centered design and evaluation
3.1 Introduction to human-centered research . . . . . . . . . . . .
3.2 Methods of enquiry . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Challenges in usability research . . . . . . . . . . . . . . . . .
3.4 Conclusions and approach of this thesis . . . . . . . . . . . .

57
59
64
81
88

4 Interviews on Corpus Working Procedures
91
4.1 Aim of the interviews . . . . . . . . . . . . . . . . . . . . . . . 93
4.2 Setup of the interviews . . . . . . . . . . . . . . . . . . . . . . 94
4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.4 Summary of results . . . . . . . . . . . . . . . . . . . . . . . . 122
5 Questionnaire on Corpus Working Procedures
127
5.1 Aim of the questionnaire survey . . . . . . . . . . . . . . . . . 129
5.2 Survey setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5.4 Summary of results . . . . . . . . . . . . . . . . . . . . . . . . 191
7

8

Contents

6 User requirements towards corpus tools
195
6.1 Demands concerning corpus resources . . . . . . . . . . . . . 197
6.2 Demands regarding general aspects of tools . . . . . . . . . . 198
6.3 Demands in corpus analysis procedures . . . . . . . . . . . . . 199
6.4 Demands related to best practices . . . . . . . . . . . . . . . . 202
6.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
7 Outline of extensions for corpus analysis tools
205
7.1 Innovation potential for corpus tools . . . . . . . . . . . . . . 207
7.2 Documentation, consulting and training . . . . . . . . . . . . 209
7.3 Technical robustness and usability of tools . . . . . . . . . . . 210
7.4 Support for workow and data management . . . . . . . . . . 211
7.5 Functionalities for processing and analysis . . . . . . . . . . . 213
7.6 Linguistic visualizations for corpus tools . . . . . . . . . . . . 225
7.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
8 Conclusions

249

Bibliography

253

A Interview data

273

B Questionnaire data

279

C Visualization examples

321

D Sample data and 'R' scripts

333

-

Chapter

1-

Research outline

910

Chapter 1. Introduction

11

Corpus linguistics (CL) started to establish as a linguistics researchmethodology in the late 1950ies and by today it has become one of the major approaches for applied linguistic research, with a myriad of application
contexts (see Chapter 2). As an empirical approach to linguistic study
it relies on authentic language data which is compiled into electronic corpora. The corpora serve as basis for linguistic analyses, which are carriedout based on automatic or semi-automatic procedures. Due to the prevalently large amounts of data in modern corpora, which make their manual
analysis infeasible, we observe a strong interdependency between corpus linguistics methodology and the employment of software tools
for corpus analysis (cf. Section 2.2, and i.e. Biber et al. (1998) and Anthony (2013b)). Anthony (2013b, p. 141) is very clear when reminding that'it is important to recognize that corpora are simply linguistic data and []specialized software tools are required to view and analyze them'.

Indeed,automatic procedures are an integral part of everyday corpus linguistics research practice, and as Anthony (2009, p. 87) observes '[w]ithout corpus tools
in the form of concordancers, word frequency counters and collocate prolers,many of the actions that we usually complete in seconds would take years of
work'.

McEnery and Hardie (2012, p. 27) further back-up this observation
by stating that 'the set of questions that may be readily addressed using a
corpus grows signicantly as suitable tools for interrogating that data become
available'.

Given the crucial role of computational software tools, it comessomewhat as a surprise that corpus linguistics still mostly relies 'on methods of observation that were developed at its conception' (Anthony, 2009, p.
87), namely 'a rather limited set of tools that mostly oer identical features'(p. 89). Standard functionalities of corpus analysis software include concordancing, collocations, frequency lists, keywords, thinning, sorting, frequency
distribution and dispersion (see Section 2.5.2.3).

Yet as of today, decisive
advancements in corpus software are mainly found on the technical andarchitectural level (i.e., related to speed, scalability, support for various
data formats, encodings and complex annotations) and more recently also
in relation to advanced support for statistics and visualizations as well
as specic innovative features in single tools (see Section 2.5.3 and 7.6.3, cf.
Rüdiger (2018); Gomide Rodrigues (2020); Machálek (2020); Brezina et al.
(2020)).
Dierent explanations could be given for the limited innovation andlargely stagnation of corpus tool advancements. On the one hand,
the limited market value and consequently non-commercial context ofcorpus tool developments, paired with a high variety and specializationof objectives in the use of corpus tools, might have hindered wide reaching developments. On the other hand, the interdisciplinary gap between
tool developers and tool users and the lack of systematic user studies
could have posed another major obstacle for progress in corpus tool develop-

12Chapter 1. Introduction

ments. In fact, Anthony (2009, p. 88/89) observes that 'computer software
engineers are notoriously poor at creating software for a purpose they do not

fully understand, and for an audience they are unfamiliar with' and suggests

to encourage corpus linguists to play a major role in the creation of corpus
software, in order to address the current situation. Also, Mason (2008, p.
144) warns that:'[D]eveloping software for exploring new methodology [is] a nontrivial task, limited to those few corpus linguists who are either
self-taught programmers or who originate from other disciplines(e.g. computational linguistics). This can seriously stie innovation, as most research relies on existing implementations ofwell-known procedures, often with severe limitations.'

In fact, little systematic knowledge or formal typologies on users,
their needs and working procedures, as well as knowledge about theusability and utility of tools for corpus analysis are yet available inthe eld (cf. Section 2.6 and e.g., Santos and Frankenberg-Garcia (2007),
Anthony (2009)). While theoretical aspects of the corpus linguistics approach and its various use cases and application areas are well documented
and discussed (cf. Chapter 2 and e.g. Leech (1991), Sinclair (1991), Teubert (2005), Partington (2006), Gries (2009)), we observe that conditionsof actual corpus use have yet been researched to a smaller extent and'user-centered evaluation with respect to corpora remains scarce' (Santos andFrankenberg-Garcia, 2007, p. 338).
Building on the understanding of McEnery and Hardie (2012, p. 37) thatthe potential of corpora will only be unlocked once linguists will be able to
make use of tools 'to manipulate and interrogate corpus data in linguisticallymeaningful ways', this thesis aims to report user-centered research on the
linguistically-oriented needs of corpus users towards software tools
for search and analysis of corpus data. By doing so it aims to incentivize
new functionalities for corpus tools or extensions of existing ones.
1.1

Status quo of user-centered research in corpus linguistics

Several linguists and corpus tool developers have explicitly raised the critical demand for user-centered research in corpus linguistics (cf. Santos
and Frankenberg-Garcia (2007); Anthony (2009); Gomide Rodrigues (2020)).
The demands particularly relate to the usability of tools and their functionalities and exibility, or to the need for a better understandingof the corpus users.

1.2. Research objective and approach

13

Despite the long standing demand for user-centered research, yet only
little related works can be found on the topic (see Section 2.6). The few
works found, focus on user needs in corpus-based language teaching,small-scale requirements or usability analyses within the scope of tool
developments and feature-oriented system evaluations of existing software tools. Related works are discussed in detail in Section 2.6.
1.2

Research objective and approachThis thesis aims at delivering new insights on requirements towards corpus analysis tools in order to foster the development of advanced tool
functionalities that respond to the actual needs of the corpus users. The
research starts from the assumption that the interdisciplinary gap between
the tools' users1 and the tools' developers2 has widely hampered major advancements in corpus software for the past decades, a concern that is sharedby researchers in the eld (see above and cf. Mason (2008); Anthony (2009)).
Accordingly, we assume user-centered investigations on needs in corpus linguistics to be a key requisite for devising concrete and applicabledirections for substantially advancing corpus analysis tools and related working procedures. In particular, concern with researching the practical implementation of corpus linguistics from the users' perspective promises
to uncover complementary information to theoretical examinations ofcorpus linguistics needs and objectives found in the literature.
In order to complement the theoretical groundings of the discipline witha usage-based view, the research of this thesis is approached by means of auser-centered study on corpus linguistic research practice. Indeed, corpus
linguists are users of methods and tools3 for corpus analysis, and domain
experts of their linguistic research elds. Accordingly, they are the mostdirect channel to understanding corpus linguistics research practice.
Thus, principles and methods from user-centered research and usability
engineering are expected to oer relevant and meaningful methodologies
also for the corpus linguistics context (cf. Chapter 3).
All parts of the presented user study are carried out in an exploratory
fashion, as core parameters for researching and evaluating corpus working
practice are not yet consensually established and can be challenging to determine (see above and cf. Section 3.3). In particular, the categories fordescribing and structuring the information bits that are relevant for determining user requirements still need to be dened in a systematic way. In
1linguists, who are using corpora
computer scientists/computational linguists, who are managing the corpus data andare developing the tools for their analysis
3Here, the term user is not limited to users of a concrete software tool, but understoodas users of the methodological approaches or techniques of corpus linguistics.
2

14

Chapter 1. Introduction

the presented study, this is done by linking ndings from the user investigations to the theoretical groundings of the corpus linguistics domain,
thus identifying the major building blocks of corpus linguistics research with focus on analysis procedures. Approaching the subject in an
exploratory fashion allows to progressively identify, dene, adapt and extend relevant categories of description in order to build the groundwork of aframework of reference which mirrors the observed reality while allowing
to accommodate key information and new ndings on the way.
More specically, the study is carried out by means of three consecutive user-centered investigations. The investigations sequentially build
one upon another, moving from two sets of open-ended exploratory investigations in the form of interviews to a quantitative investigation of speciedsubjects in the form of a questionnaire survey. Each investigation serves
to validate or adjust results of the preceding study, and to inform the subsequent one. The continuous study design allows to start from a highly open
and wide perspective which is successively narrowed down to detail relevant selected aspects and nally evaluate them from a quantitative
perspective.
As the rst and second investigation, interviews are carried out withmostly experienced corpus users. First, a series of open interviews (I1)enquiries about workows of concrete corpus studies and the usage of tools.
It aims at gaining a general overview of corpus working practice and atdetermining its key aspects. Second, a series of focused interviews (I2)
addresses in detail those key aspects that are closely related to corpus analysis procedures. In contrast to I1, the I2 series abstracts away from concrete
corpus studies and workows and takes a factorized and higher-level view on
the aspects of interest. It aims at collecting detailed information on eachcategory derived from I1. The setup and results of both interview series arereported in Chapter 4. As the third and nal investigation a structured
questionnaire survey (QS) reproposes the intermediate ndings of thetwo interview series to a larger group of corpus users in a structured way.
The questionnaire survey, primarily aims to quantify instantiations of thedetermined categories, validate their plausibility and coverage, and evaluate interdependencies between user characteristics and selected aspects ofcorpus research procedures. The setup and results of the questionnaire
survey are reported in Chapter 5. For each of the three investigations theobjectives and methods of investigation are summarized in Table 1.1.
Finally, based on the gathered observations, open demands for corpus
analysis tools and procedures are determined (cf. Chapter 6) and possible ways to address the requirements on dierent levels of abstraction
are discussed, including suggestions for prospective tool functionalities (cf. Chapter 7). An overall summary of the presented work and its
ndings and conclusions is given in Chapter 8.
The results of the study and thus the contribution of this thesis is referred1.3. Scope and timeliness

15

Table 1.1: Objectives and methods used in the three steps of the user study
Objective
Interviews 1 (I1) Gain overview of corpus
working practice and determine its key aspects
Interviews 2 (I2) Detail aspects related tocorpus analysis procedures
Questionnaire
survey (QS)

Quantify and expand selected feature dimensionsof corpus analysis procedures, and evaluate interdependencies among them

MethodCase-based investigation
on workows as open (expert) interviews
Abstract
investigation
on factorized aspects ofcorpus analysis as semiguided expert interviews
Quantitative investigation
as structured questionnaire on dened categorysets

to corpus users as well as developers of corpus software tools.
1.3

Scope and timeliness

The user-centered study investigates on open demands for corpus use inlinguistics research and related computational support, while corpus
uses for other purposes4 are not considered. Accordingly, the study exclusively targets users who carry out linguistic studies based on corpora andexcludes any other type of corpus users. No restrictions to specic subjects
in linguistics have been applied. Furthermore, the study is focussing on needs
and research practice regarding the human-steered analysis of corpora(cf. Section 2.4.2), as opposed to corpus building (cf. Section 2.4.1), including corpus processing (e.g. manual or automatic annotation, etc., cf. Section
2.4.1.3 and 2.4.1.4). The research is limited to work with corpora of written
language while the range of languages has been kept open.5
Methodologically, the user study is based on explicit investigationsof corpus users, while no eld studies or walk-through observations are carried out (cf. Section 3). Individual experiences of overall 128 corpus users
have been collected by means of two interview investigations and a questionnaire survey.
4e.g., corpus analyses for translation or teaching purposes
Even though no language limitation was imposed, for the interviews European andmainly German study participants were involved, and the results showed (both for theinterviews and questionnaire survey) that most of the collected insights relate to the workwith European languages.
5

16

Chapter 1. Introduction

Results of the interviews date back to 2007 and the questionnaire survey was carried out in 2012/2013. Despite the considerable amount of timethat has passed since the user investigations were conducted, the insights
are still remarkably up-to-date. Indeed, to our knowledge no comparable
study of similar scale has been published meanwhile and overall very little
research on user-centered aspects of corpus linguistics or the evaluation of
corpus tools has been performed since the beginnings of the work presented
in this thesis (see Section 2.6).
At the same time, recent publications like Knight et al. (2020) attest that'[w]hile there is no one strict 'standard' approach to corpus analysis (as it
depends on the specic research questions at hand), there are commonalities
in the suite of analytic tools available, which commonly include search, sort,word list, concordance, keyword and n-gram functionalities'. This illustrates

a current reality, which this thesis aims to address. A reality, in whicha wide variety of dierent research questions calls for dierent approaches
to address them, but software tools mainly cover a standard set of basic
functionalities that have been available (even though in less rened forms)
since the early days of modern, that is computer-based, corpus linguistics(cf. Chapter 2, Section 2.5.3).
The continuing near-to-stagnation of functional innovations in corpus software tools paired with a persistent scarcity of user-centeredresearch in corpus linguistics is remarkable. It amplies the call for more
research on the needs and working realities of corpus users as investment into identifying relevant innovative options for new tool features.
On the one hand, taking a closer look at the users' needs should accelerate
the dynamics of software tool development in the future. On the other
hand, it could bring important insights on general stumbling blocks
that might be out there and would have to be overcome rst.
This thesis aims to make a rst step towards a better understandingof the users' perspectives and realities and a formalization of thefactors and dimensions that are relevant building blocks in corpus linguistic analyses. By doing so, the thesis does not only aim at delivering a better
understanding of the current situation and possible indications for tool
developments, but it also aims at nudging further interest and researchin the direction of user-oriented design and development of softwaretools for corpus analysis.
1.4

Structure of the thesis

This introductory chapter has outlined the overall research objectiveof this thesis. It has detailed which research gap it aims to close, has pointed
to the existing related work and has indicated the scientic approach taken
for pursuing the research.

1.4. Structure of the thesis

17

The following two chapters provide the relevant background for thisresearch. Chapter 2 gives detailed information on the domain of corpus linguistics and discusses its current status in relation to the research
objective. Chapter 3 introduces relevant principles and methods of usercentered research and thus provides the methodological grounding for the
research study.
Chapter 4 and 5 describe the user studies that were performed forthis thesis and thus constitute the core part of this thesis. Chapter 4 isconcerned with all aspects of the interview investigations, while Chapter
5 reports in detail on the questionnaire survey study. For both studies theaims, setup, results and conclusions are presented. Chapter 6 summarizes
the list of demands that result from the interview and survey investigations
in a concise fashion.
Chapter 7 exemplarily outlines possible responses to the demands
on dierent levels of abstraction, including a short discussion of the potential of visualizations as extensions to corpus tools. Finally, Chapter
8 concludes this thesis by summing up its overall ndings, contributions
and next steps for future work.
In addition, four annexes list supplementary material related to the user
studies (interviews A and questionnaire B), illustrative samples of visualizations for language data (C) and relevant excerpts of the scripts for the
analysis of the questionnaire data D.

18

Chapter 1. Introduction

Figure 1.1: Thesis structure-

Chapter2-

The corpus linguistics research domain

19

20

Chapter 2. Corpus Linguistics

2.1. History

21

Aim of this chapter

This chapter provides an overview of the corpus linguistics research approach. It serves as grounding for the research of this thesis and aims at
explicating its frame of reference.
Relevant theoretical and practical aspects of corpus linguistics are summarized and discussed. The historical development of corpus linguistics
is summarized in Section 2.1. Section 2.2 details the objectives of the corpus
linguistics approach and discusses its fundamental theoretical principles.
An overview of dierent application areas of corpus linguistics is given
in Section 2.3. Finally, practical research methodologies are laid out in
Section 2.4, and data and tools are described in Section 2.5. The current
state of user and usability research in corpus linguistics is discussed in
Section 2.6. The chapter closes with a short conclusion and recapture of
the aims of this thesis in Section 2.7.
2.1

History

Attempts in collecting language data as empirical basis for language
studies are reported as early as the 17th century (Kennedy, 1998). Lexicographers based their work on collections of citations from renowned writers, and large concordances were compiled in the context of biblical studies.
Around the beginning of the 20th century corpora started to also get employed for grammar building (Damascelli and Martelli, 2003, p. 15).
However, the beginnings of modern corpus linguistics, as we understand it nowadays, date back to the late 1950s only. Advances in computerand information science had started to allow for pursuing data-driven linguistics on a larger scale, and a strengthened focus on researching language
in use (cf. Figure 2.1) promoted an empirically-based approach to thestudy of language.
The developments of that time happened concurrently around several
research groups in Northern Europe and the United States. At Brown University, Henry Ku£era and W. Nelson Francis compiled the very rst computerized corpus, the Brown Corpus of American English (Francis and Ku£era,1964), in the early 1960s. It was created with the aim to provide a basis forcomputational analysis and lexicographic studies.
Almost at the same time, starting in the late 1950s, at the University
College London a group of scholars worked on the collection of data for
the Survey of English Usage (SEU) (Quirk, 1960), which set the groundworkfor the Lancaster-Oslo/Bergen Corpus (LOB) of written British English (Johansson et al., 1978) and later on the London-Lund Corpus of spoken British
English (Svartvik, 1990).

22

Chapter 2. Corpus Linguistics

Figure 2.1: 'Language in use' and the Firthian tradition
Language in Use
By building on proofs of language in use the corpus linguistics approach connects to the Firthian tradition that aims
at understanding language in its situational context (Firth,1957). By analyzing 'attested language' it stands somewhat
in opposition to more cognitively oriented approaches to thestudy of language, which have been promoted in particular
by Noam Chomsky (Chomsky, 1962, 1965, 1968).

In the early 1970s, another major branch of corpus linguists evolved at
the University of Birmingham around John Sinclair (cf. e.g. Sinclair (1991)).
Theoretically rooted in J.R. Firths social and communicative account of language (Firth, 1957), it gave particular attention to the relevance of studying'attested language' in context (cf. Figure 2.1). The COBUILD corpus ofwritten and spoken British English (Renouf, 1987) is the rst corpus createdin this context.
Since its beginnings in the late 1950s to the early 1960s, corpus linguistics has greatly developed and expanded in terms of linguistic application
contexts, research community, and resources. In its beginnings, corpus
research addressed a small set of linguistic research areas including lexicography, language acquisition and grammar theory, and prevalently focused on
the English language. From the 1980s onwards corpus studies have boomed(McEnery and Wilson, 2005) and by today the corpus linguistics approach isapplied to several research elds within and beyond linguistics (cf. Section2.3), and covers a wide variety of languages, including minority languages,
language varieties and dialects (cf. e.g. Wilson et al. (2002)).
Accordingly, the corpus linguistics research community has been
growing worldwide. This is well reected in the increasing number of related publications, the founding of dedicated journals1 , book series2 and theestablishment of recurrent conferences3 , as well as the range and size ofavailable corpora (see Table 2.1), and the maturity and variety of tools
1

E.g. the International Journal of Corpus Linguistics (John Benjamins Publishing
Company) was launched in 1995 and Corpus Linguistics and Linguistic Theory (Mouton
de Gruyter) in 2005.
2
Studies in Corpus Linguistics (John Benjamins Publishing Company) since 1998, and
Edinburgh Textbooks in Empirical Linguistics (Edinburgh University Press) since 1996
3
E.g. the biennial 'Corpus Linguistics Conference' organized by the University Centre
for Computer Corpus Research on Language of Lancaster University (UCREL) was initated in 2001 and since 2009 the 'International Conference on Corpus Linguistics' (CILC)
is organized annually by the Spanish Association for Corpus Linguistics (AELINCO).

2.2. Objectives and theoretical principles

Time
First
generation
corpora(60ies/70ies)
Secondgeneration
mega-corpora(80ies/90ies)
21st century
corporaWeb corpora

Corpus nameBrown Corpus of American English(1967)
London-Lund Corpus, spoken (1975)
Lancaster-Oslo/Bergen Corpus (LOB)(1979)COBUILD corpus (1987)
British National Corpus (BNC) (1994)
International Corpus of English, spoken (ICE) (1998)
La Repubblica corpus of Italian (2004)
German DWDS core corpora (2010)
Corpus of Contemporary American English (COCA) (2015)
British Bank of English (current )Cambridge English Corpus (current )Wacky corpora (ukWaC, deWaC,
itWaC) (2008)
TenTen (1010 ) web corpora (2013)
Global Web-based English (GloWbE)(2013)
COrpora from the Web (COW) (2016)
Oxford English Corpus (current )23

Size in tokens
1 million
435,000
1 million
20 million
100 million
1 million
380 million
136 million
520 million
650 million
multi-billion
1-2 billion
10 billion
1.9 billion
10-20 billion
2.5 billion

Table 2.1: Increase of corpus sizes over time
for corpus processing and analysis (cf. Section 2.5).

2.2

Objectives and theoretical principles

Corpus linguistics is concerned with 'the study of language based on ex-

amples of 'real life' language use' (McEnery and Wilson, 2005, p. 1). Moreconcretely it is 'an exploration of language based on a set of authentic texts
in machine readable format' (Gatto, 2014, p. 7), and thus an empirical approach to linguistic study which relies on collections of authentic language
examples assembled in corpora.
In terms of its practical scope and application context corpus linguistics is bound to the analysis of language in use. It does not genuinely target
the derivation of knowledge on linguistic competence and neither on the human language faculty and language-related processes in the human mind (cf.
Chomsky (1965, 1968)). As stressed by Lemnitzer and Zinsmeister (2010,

24

Chapter 2. Corpus Linguistics

p. 15), corpus linguistics is a fundamentally 'purpose-driven'4 approach,which aims at achieving measurable, generalizable, and replicable results on
language-related research questions. This is enforced by following a systematic and replicable methodology. In fact, corpus linguistics displays astrong concern for methodological questions. As Kennedy (1998, p. 271) explains '[c]omputer corpus-based approaches to language use well-establishedscientic procedures involving observation, analysis, theory building and subsequent verication'.

And McEnery and Wilson (2005, ch.7) stress that an'emphasis on the systematic encoding of data and tools for its manipulation
is an area where corpus linguistics excels'. However, standards and practices

regarding methodology continue to mature up to today. While in early years
Kennedy (1998, p. 206) had stated that 'the methodologies which best exploit
the potential of corpora are still not as well established as is needed ...' (p.
272), also several years later Lemnitzer and Zinsmeister (2010, p. 10) still
stress that 'scientic standards for corpus linguistics partly still have to be
developed'. Furthermore, corpus linguistics strongly relies on automatizedprocedures and computer tools, in order to support the eciency of applying the devised methods on large amounts of data (large corpora). In
particular, for tasks related to corpus building and corpus analysis a rangeof dierent tools are available and widely used (cf. Section 2.5.2).
These key aspects of corpus linguistics are reected by Biber et al.
(1998, p. 4) in their description of corpus-based analyses along four characteristics: (1) 'being an empirical approach', (2) 'being based on representative
corpora', (3) 'relying on computer software' and (4) 'making use of qualitativeand quantitative analytical techniques for interpretation'. Tognini-Bonelli(2001, p. 2) adds a fth characteristic related to the situational context:(5) 'operating within the framework of a contextual and functional theory of
meaning'.
The remainder of this section describes the major theoretical principles which stipulate scientic standards and best practices of the corpus
linguistics approach.

Linguistic theory and empirical evidence
A key question in corpus linguistics is how to balance linguistic theory andempirical evidence, and whether codied theoretical knowledge or empirical data should be the starting point of the analysis process. Historicallythis opposition is known as the corpus-based vs. corpus-driven approachto corpus linguistics (see Figure 2.2).5
The theory-steered (a.k.a. corpus-based) view postulates that a corpus study should start from theoretical assumptions, which are derived from
4

German (as original): 'zweckorientiert'
In computational terms this opposition translates to the 'top-down' vs. 'bottom-up'
approach in relating corpus data and linguistic theory.
52.2. Objectives and theoretical principles

25

introspection or other corpus-independent sources and are explicitly formulated as linguistic hypotheses. The corpus study serves to conrm, reject orwiden the theoretical hypothesis based on corpus evidence. In this perspective corpus linguistics is understood as a method for validating theory bymeans of empirical data.
The data-steered (a.k.a. corpus-driven) view claims that the primaryvalue of corpus linguistics is tributary to the fact that corpora can reveal
new and unexpected aspects of the language under study. Accordingly, thisperspective demands that empirical corpus data is analyzed without any
theoretical pre-assumptions, which potentially could bias or misconstrue the'pure' evidence from data. Empirical data takes precedence over theory, andis de facto considered the basis for any theory-building.6 In this perspective
the corpus is taken to have 'theoretical status' (Tognini-Bonelli, 2001, p. 1).
Historically the opposition between theory-steered and the data-steered
approaches has strongly been argued within the community (cf. e.g. Fillmore (1992); Teubert (2005); Hardie and McEnery (2010); Gries (2010)).
Nowadays the distinction is less marked, and in practice the approaches areprogressively converging and mixing (Hardie and McEnery, 2010). In addition, corpus linguistic approaches increasingly get combined and integrated
with other means of linguistic enquiry, like eld studies or experimental
methods (Gilquin and Gries, 2009).

Validity and generalizability of ndings
The validity and generalizability of linguistic insight derived from corpora depends on the representativeness of the underlying corpora for the
language they stand for. Safeguarding the quality and adequacy of a corpus
as empirical basis for a given research purpose, is a key challenge of corpus
linguistics.
Creating representative corpora is enforced by a careful corpus design(cf. Section 2.4.1), which targets a balanced sampling of valid authentic texts
for a corpus of adequate size. However, due to the nite nature of corpora, as
well as due to fundamental and pragmatic constraints for creating perfectly
balanced copies of a (sub-)language, a corpus cannot be fully representative,
and therefore also absolute generalizability cannot be assumed.
In addition to the limitations of representativeness, a corpus is always
limited in scope. The explanatory power of insights gained from a corpus
remains restricted to the language that the corpus stands for. This is inaccordance with the substantial presumption of the corpus linguistics approach, to study language in its textual context (cf. Section 2.2) and its
situational context in terms of its linguistic and social dimension.
6

In the data-steered view, corpus data may fundamentally question and redene established theoretical stances or categorizations (like e.g. the typology of word classes orthe established distinction between lexic and grammar).

26

Chapter 2. Corpus Linguistics

Figure 2.2: Two traditions in corpus linguistics
The corpus-based and the corpus-driven approach
Historically, corpus linguistics has emerged and developed in parallel withindierent schools (cf. Section 2.1), which formed two major methodologicalperspectives: the corpus-based and the corpus-driven approach.
The corpus-based approach, rooted in the works on the Survey of English Usage (Quirk, 1960) in the late 1950s, understands corpus linguistics asa tool or method which makes use of empirical language data (assembled incorpora) for validating linguistic theories, which in turn are derived fromtheoretical assumptions, introspection or other corpus-independent sources.
Hardie and McEnery (2010, p. 386) dubbed this approach to corpus analysis
the 'corpus-as-method' or the 'methodologist' tradition, as it 'views corpus
linguistics as a method for the analysis of language' (p. 385).
The other tradition, the corpus-driven approach, is rooted in the workof J.R. Firth, continued by J. Sinclair, and has been developing at the University of Birmingham in the early 1970s (Tognini-Bonelli, 2001, p. 157 .).
It gives primary weight to empirical data over theory and argues that corpus evidence should drive the entire linguistic research process,including the building of analysis categories. Hardie and McEnery (2010, p.
386) propose the terms 'corpus-as-theory' or 'neo-Firthian' tradition, as 'thetradition rooted in Sinclair's work, views corpus linguistics as a sub-eld in
its own right' (Hardie and McEnery, 2010, p. 385).

The representativeness of corpora is a key objective of the corpus designphase (cf. Section 2.4.1), and is equally relevant for the analysis process (cf.
Section 2.4.2) in which the corpus characteristics (sampling, metadata, etc.)
have to be taken into consideration.

Qualitative and quantitative information
Corpora provide both qualitative and quantitative information on linguistic structures (cf. Lemnitzer and Zinsmeister (2010, ch. 6.3.2)). Accessto quantitative information is a key strength and unique feature of the
corpus linguistics approach, when compared to other approaches in linguistics7 , and, given certain preconditions, allows for the statistically valid generalization of ndings. At the same time, the explanatory strength of qualitative data can go beyond the explanatory power quantitative data. For
7

While native speakers are able to provide grammaticality judgements for linguistic
structures, 'language users are often not consciously aware of their most typical choices'(Conrad, 2010, p. 227); cf. also Alderson (2007).

2.2. Objectives and theoretical principles

27

example, lists of examples in context oer a comprehensive and ne-grained
perspective on the data, while quantication always entails classication,and thus reduction (McEnery and Wilson, 2005).
This situation leads to methodological discussion on the balancing between qualitative and quantitative analyses and the question of how to combine, weigh and relate qualitative and quantitative data in the best possible
ways.

Corpus linguistics and computer tools
As Kennedy (1998, p. 268) recognizes, 'the ability to use computers to nd,sort, analyse and quantify linguistic features had a considerable impact on
the development of corpus linguistics'.

In fact, corpus linguistics research
practice strongly relies on the use of computers for the processing andanalysis of data, and 'it is important to recognize that corpora are simply
linguistic data and that specialized software tools are required to view andanalyze them' Anthony (2013b, p. 141). Software tools are employed for the

linguistic processing of corpus texts (cf. Section 2.5.2.1) as well as for the
search and analysis of corpora (cf. Section 2.5.2.3). Therefore, '[t]he successof the [corpus linguistics] approach is intrinsically related to the tools used toaccess, analyze, and display the results of corpus searches' (Anthony, 2013b,

p. 159). Tognini-Bonelli (2001, p. 5) summarizes the overarching role ofcomputer power by stating that corpus work makes use of and is madepossible by new technologies. At the same time, McEnery and Hardie (2012,
p. 36) point out that while being a powerful aid, 'tools for exploring a corpus[] also, crucially, limit and dene what we can do with a corpus'.
Given the fundamental relevance of computer tools, a methodologicalchallenge for corpus linguistics is to dene how tools can best support corpus
research, and, at the same time, to monitor and evaluate to what extent
technology shapes methodology. The high impact of tools and the need toact on it is addressed well by the statement of Anthony (2013b, p. 141):'The functionality oered by software tools largely dictates what
corpus linguistics research methods are available to a researcher,and hence, the design of tools will become an increasingly important factor as corpora become larger and the statistical analysis
of linguistic data becomes increasingly complex.'

Raw corpus and annotations
The question how much linguistic knowledge should be encoded into a corpus
is an issue in corpus linguistics, which is related to the balancing between'linguistic theory and empirical evidence' (see above). Discussions concern
the question what degree of annotation, if any, is methodologically most

28

Chapter 2. Corpus Linguistics

favorable in terms of the explanatory power of the corpus as empirical research basis. The strictly data-steered (corpus-driven ) approach favours a
raw corpus without any annotations (text only). Any type of annotation
is assumed to introduce a bias and thus to negatively aect the genuine explanatory power of a corpus. More drastically put, 'the strongest argument
against annotation is that it 'contaminates' the original data making it moredicult to see language patterns' (Anthony, 2013b, p. 147).

The theory-steered (corpus-based ) approach considers annotations generally to be a benet. They are assumed to add relevant information andto allow for advanced modes of searching in a corpus. On the practicallevel, however, there is less need to make a denite decision on this question,
as '[m]odern corpus tools are easily able to show or hide dierent layers ofannotation or markup of texts' (Anthony, 2013b, p. 148).

Quality of data and reliability of tools
Reliable data and tools are a crucial prerequisite for the validity of any
analysis. Errors in the raw data and annotations can negatively impact research results.The same problem results from poorly performing or unhandy
tools for text processing and analysis. In the case of annotation tools, theirlimitations transmit to the corpora processed with them, for example byproducing unreliable linguistic annotations. In the case of buggy analysis
tool searches might not return the results that a user expects. Also, tools
which are particularly dicult or unintuitive to use are likely to cause errors
through maloperation by the user.
More generally, Anthony (2013b) points out that the analysis tool itself
does have an eect of the data that a user will receive during his analysis.
For example, the way a tool determines word boundaries will impact onword counts. Accordingly, Anthony (2013b, p. 153) warns that 'it is easy
for researchers to forget the ltering eect of the tool and begin using it in
an unquestioning manner'.

Systematic and replicable research methodology
Finally, the denition and enforcement of a scientically valid methodology for data processing and analysis is a general challenge for corpus
linguistics  as for any empirically based scientic discipline.
In fact, the replicability of procedures and the reproducibility ofanalysis results as well as the possibility to trace back the impact of certain
processing decisions is taken as a fundamental criterion for the scienticvalidity of analyses (cf. e.g. Lemnitzer and Zinsmeister (2010)).
Corpus linguistics best practices consequently call for the systematic documentation of data and analysis procedures. This includes the documentation of design decisions, text processing and annotation steps involved in2.3. Linguistic application areas

29

corpus building (cf. Section 2.4.1) as well as the methods and overall workows used in corpus analysis (cf. Section 2.4.2).
Furthermore, systematic and replicable research is enforced by formalizing procedures, and by increasing eciency through the adoption of automatized processing procedures and the employment of computer tools for data
storage, access and analysis. Standards for text encoding (cf. e.g. TEI8 ) andannotation formats (cf. e.g. PAULA (Dipper, 2005)) as well as processing
frameworks (cf. e.g. UIMA9 ) and attempts to provide infrastructures fortool chains (cf. e.g. WebLicht (Hinrichs et al., 2010), Switchboard (Zinn,2016)) give examples of the practical implementation of the formalization ofprocedures.
2.3

Linguistic application areas

Initially the corpus approach was prevailingly employed in lexicographic
studies, historical linguistics and language acquisition research (Lemnitzer and Zinsmeister, 2010). Over the years, its application context continuously expanded to a wide range of linguistic subjects (cf. e.g. Baker (2009);
Sampson (2004)). In fact, as McEnery and Hardie (2012, p. 26) observe that'it is now [] dicult to nd an area of linguistics where a corpus approach
has NOT been taken fruitfully'.

Below several major application areas arebriey sketched.
In addition to linguistic studies, corpora also have a tradition in language teaching (Sinclair, 2004; O'Keee et al., 2007; Aijmer, 2009) andtranslation studies (Baker et al., 1993; Fantinuoli and Zanettin, 2015),
as well as for the development of NLP tools in computational linguistics(Church and Mercer, 1993; Carstensen et al., 2009). To a smaller extent, corpus methods are also employed in literary studies, stylistics (including e.g.
academic writing), sociolinguistics, studies of political and media discourse,
forensic linguistics, etc. (see O'Keee and McCarthy (2010)).

Grammar and syntax
The study of syntactic structures and dierent kinds of grammatical relations is the classic application area of corpus linguistics. Using empirical
corpus data for deriving and testing hypotheses on linguistic constructions,serves theoretical linguistics as well as grammar building for language teaching. The analysis of grammatical structures is purely text internal with local
scope (e.g. within a sentence). In fact, analyses on the syntagmatic level area subpart of many investigations that are concerned with the overall study
89

http://www.tei-c.org/index.xml
https://uima.apache.org/

30

Chapter 2. Corpus Linguistics

and description of a type of language or a language variety, as are describedin the following subsections.

Discourse analysis and pragmatics
Corpora are used for analyzing the semantics and pragmatics of sentences or discourse units (e.g. turn taking, distinction of speech acts, etc.)
and for studying discourse structure. In contrast to syntactic studies, forstudies in semantics and pragmatics the unit of investigation often exceeds
sentence boundaries, and in case of discourse extends to entire texts. The useof corpora for the analysis of media/political discourse has recently gained
increased attention. Virtanen (2008) distinguishes two aspects of discourse:(1) discourse as the comprehensive human communication with a focus on
the linguistic, situational and cultural context of language, and (2) discourse
as the evolution of text or conversation with a focus on aspects like coherence,cohesion, and discourse markers.

Lexical studies: Collocations, phrases, multi-word expressions,lexical semantics, metaphors
Corpora are employed for the study of lexical semantics, collocations,multi-word expressens, phraseology and metaphors. Research on collocations and related concepts has a long history in corpus linguistics, oftentimes as part of lexicographical works. Investigations on the habitual cooccurrence of words and resulting implications on their meaning ('You
shall know a word by the company it keeps' (Firth, 1957)), as well as studies on multi-word expressions or phrases and their degree of xedness
are typical research subjects in this area, with focus on lexico-grammatical
relations in language. Distributional semantics is another closely related
research topic.

Register and genre studies
Corpora are employed in genre and register studies to analyze linguisticfeatures in relation to dierent text categories. When concerned with text
genre the focus is on linguistic characteristics which are used to structure
texts, while registers of text are distinguished by pervasive linguistic features,
which are typical and distinctive for dierent text varieties (Biber, 2010).
According to Biber (2010), register studies can be grouped into three
types: the study of a particular linguistic feature and its variation within
one or across several registers, the study of linguistic characteristics of aparticular register, and comparative analyses of the linguistic characteristics
of dierent registers. In addition, genre studies analyze and document thecharacteristic rhetorical organization of dierent text categories.

2.3. Linguistic application areas

31

Study of language varieties and dialectology
From the early days of modern corpus linguistics, research has been doneon the description and distinction of language varieties and dialects. Already based on the LOB and Brown corpora a word frequency comparisonof American and British written English has been conducted (McEnery andWilson, 2005, p. 125). The rst large-scale project on varieties of English
has been initiated with the compilation of the International Corpus of English (ICE) (Greenbaum, 1996). Nowadays, corpora for varieties of many
languages exist (cf. e.g. Dittmann et al. (2012) for a corpus of German varieties). To the contrary, dialect corpora are yet less frequent. Traditionally,
dialect data is assembled in dialect atlases and information about dialects
is collected via questionnaires rather than researched in language corpora.
This is also due to the fact that dialect is mainly manifested in spoken language and the denition of writing conventions is one of the many obstacles
for dialect corpus creation.

Learner language and computer-mediated communication
Learner language and computer-mediated communication (i.e. language used in online communication, like chats, blogs and online forums)
are another two types of language varieties, which are studied with corpusbased methods. Learner corpus research has started in the late 1980s (cf. e.g.
Granger et al. (2013)), and the study of computer-mediated communication(CMC) in the early 1990s (cf. e.g. Beiÿwenger and Storrer (2008)).
Research in both elds is concerned with analyzing learner-specic or
CMC-specic features of language use, often in relation to sociolinguistic
dimensions, such as age, level of education, etc. These non-standard language phenomena include for example orthographic and grammatical errors
in learner language, and innovative abbreviations or emoticons in CMC data.

Historical linguistics and diachronic studies
Diachronic studies in general, and studies in historical linguistics inparticular are concerned with analyzing language change. By providing a
new source of evidence for the stages of linguistic development, corpus linguistics has 'enriched not only knowledge of the history of particular languages but also theoretical models of language change' (Curzan, 2008, p.
1091). In particular, corpora allow for taking into account also the factor ofvariability within historical linguistics by examining the impact of sociolinguistic factors next to the impact of the temporal dimension.
In fact, sociolinguistic dimensions are often taken into consideration when
historical corpora are analyzed to trace morphosyntactic, lexical and semantic change as well as for observing pragmatics and discourse features.

32

Chapter 2. Corpus Linguistics

First language acquisition and psycholinguistics
Research on rst language acquisition and language pathologies is primarily concerned with understanding processes of the human mind which are
underlying language production. Therefore, in language acquisition researchexperimental approaches, partly within laboratory setups, take a naturalprevalence over studies based on authentic language corpora. Nevertheless,corpora of transcribed conversations are used as complementary empiricalresources, which attest actual language use, allow for analyses of frequency
patterns, and make up for research contexts in which the collection of reliablespeaker judgements is not possible (e.g. with small children or linguistically
impaired people).
Systematic longitudinal studies collecting spontaneous child language
started to appear in the 1960s and 1970s (cf. Diessel (2008, p. 1199))
and have been made available and shared on a larger scale for the rst timethrough the CHILDES database (MacWhinney and Snow, 1985). Before
that, child language transcripts were mainly based on individual parental
child language diaries. Highly researched subjects in this eld include theacquisition of grammatical categories as well as the acquisition of complex
sentences.

Sociolinguistics
In sociolinguistics the relation between linguistic variation and the social
context in which it occurs is studied. Sociolinguistics crucially depends on
authentic language data and metainformation on the context of its production, which should capture '[s]ociolinguistic factors [which] include the majordemographic categories of age, gender, social class and ethnicity' (Andersen,
2010, p. 547).
Sociolinguistic subjects which have been studied with the corpus linguistics methodology include gender studies, studies on the relation between
language and age (i.e. youth language), the impact of migrant's languages
on language change, etc. Sociolinguistic corpus research is often based on
spoken language data.
2.4

Research methodology

Based on the objectives and theoretical principles described in Section 2.2,
the corpus linguistics methodology aims at dening sound, systematic andreplicable procedures for empirically-based linguistic research. It is
encoded in methods and implements best practice guidelines concerning
corpus building and analysis. Accordingly, the following subsections describe
common procedures in corpus building (Subsection 2.4.1), and corpus
analysis (Subsection 2.4.2).

2.4. Research methodology

33

At the same time, the discipline has not consolidated one nal methodology for pursuing corpus linguistics research. As stated by Mason (2008,
p. 143):
Corpus linguistics is still a comparatively young discipline, andits methodology is still in ux; new procedures are developed, oldones are modied or extended, new challenges come up with an
expanding range of languages being analysed, and so forth.

Also, Lemnitzer and Zinsmeister (2010, p. 10) repeat that 'scienticstandards for corpus linguistics partly still have to be developed' and Knight

et al. (2020) give an explanation for not having 'one strict 'standard' ap-

proach to corpus analysis' that is: 'it depends on the specic research questions at hand'. This exibility and search for new and extended procedures

applies less to corpus building than to corpus analysis, which most likely
is related to the fact that corpus analyses are more varied in their objectives
and more open-ended in their targeted outcomes than corpus building tasks.

2.4.1 Procedures in corpus building
Procedures for the corpus building are well described in the corpus linguistics literature (cf. e.g. Kennedy (1998, ch. 2.6), Wynne (2005)) and typicallyconsist in ve steps10 : (1) the corpus design, (2) the collection of texts
and metadata, (3) the basic text processing, (4) the linguistic annotation of the corpus data, and (5) the data preparation for inclusion into
a querying system. In addition, the compilation of written documentation on all decisions and processing steps should systematically accompany
the corpus building process.

2.4.1.1 Corpus design
The corpus design phase serves to dene the relevant characteristics ofthe corpus and its composition in order to be representative for a languageof interest (cf. corpus denition in Section 2.5.1).
In the rst place, linguistic corpora have to be built of authentic language data, that is text produced in real-world settings, and included incomprehensive units (e.g. an entire newspaper article). Corpora haveto be composed of texts that build a representative sample of the targeted (sub)language, which is dened with respect to a specic research
objective (e.g. studying the language of young adults in a university setting). They have to be of adequate size for the purpose of the envisioned
investigation.
10
Not all of the steps are mandatory, e.g. the linguistic annotation of a corpus is not
obligatory and also metadata is often kept to a minimum.

34

Chapter 2. Corpus Linguistics

Furthermore, in order to be representative, the corpus has to provide abalanced sample of the language it stands for. That is the coverage andrelated proportions of the text types included in the corpus need to reect
their respective proportions within the language as a whole (e.g. with regard
to genre, year of publication, etc.). The dimensions along which dierent
types of corpora are distinguished are described in Section 2.5.1.

2.4.1.2 Collection of texts and metadata
The collection of corpus data consists in the selection and assembling oftexts and related metadata, which should be carried out along the corpus design plan. Nevertheless, the strict adherence to the design plan canbe impeded by common legal and pragmatic issues related to copyright, diculties with the availability of required text materials and privacy issues (e.g.
only hard-copies in low printing quality, gaps in archives, private communications, non-compliance with GDPR, etc.), or sections of the language which
are not represented by printed text (e.g. informal spoken communications,manual notes like shopping lists or blackboard announcements).
For the special case of web corpora, typically automatic tools for crawling the web and harvesting texts are used (cf. Section 2.5.2).

2.4.1.3 Basic text processing
After collecting the text documents and metadata, they often need to undergo some textual pre-processing. This can include the post-correctionof manual transcriptions or OCR-ed text, if the text documents are notreadily available in digital format. Typically also format-related operations like harmonization of encodings, transformation of le formats etc.
are needed.
Other common processing operations include the cleaning from irrelevant information (e.g. html-tags, page numbers, headers, footers, etc.),
corrections of the text (e.g. in case of corruptions during encoding conversions), the adjustment of structural formatting (e.g. mark-up ofheadings, paragraphs, etc.), etc.
The text processing phase is carried out by means of dierent types ofcomputational linguistic tools (e.g. OCR tools) and custom scripts as wellas generic tools for cleaning and conversion tasks (cf. Section 2.5.2).

2.4.1.4 Linguistic annotation
Once the corpus texts are readily processed linguistic annotations can getapplied. Annotations span all levels of linguistic description, starting fromlemma and part-of-speech information, which nowadays count as standard
annotations for most corpora, up to annotations of syntactic or semantic

2.4. Research methodology

35

information as well as, for example, particular characteristics of learner lan-

guage or computer-mediated communication, to just name a few (cf. Section
2.5.1.2 for details on annotation levels).
Depending on the type of annotation and the specic corpus building
context, annotations are applied manually or by means of automatic tools.
On the one hand, a wide variety of NLP tools (Natural Language Processing) for dierent annotation tasks and languages are available nowadays(cf. Section 2.5.2 for an overview). Performance rates of the tools vary,
mainly depending on the complexity of the tackled tasks. Partly automatic
annotation is complemented with subsequent manual processing and error
correction. On the other hand, for certain annotation tasks, manual work
is inevitable from the start. For example, this can occur, if no NLP tool isavailable, which is covering the particular use case, or if performance rates
do not live up to the required quality. Manual annotation or correction has
the downside of being time-consuming and thus cost-intensive.

2.4.1.5 Data import into querying system
In order to be automatically searchable, in a last step, the processed andlinguistically annotated corpus texts are typically imported into a search
and analysis tool (cf. Section 2.5.2.3 for examples). This often requires
further encoding and format transformations to match the requirements of
the query tool, such as initial sorting or indexing operations.

2.4.1.6 Documentation of the corpus building
To ensure the transparency and reproducibility of the corpus building
process all decisions regarding the corpus design, as well as the processing
and annotation steps have to be documented systematically. The documentation should record metadata and copyright information on the included text sources, information on the data formats and encodings, as wellas the tools used for processing (including version number, congurationand parameter settings). In addition, sustainable documentation should
ideally be accompanied by the sustainable storage of the corpus ideally inadherence to the FAIR principles11 (possibly with dierent versions of the
corpus for reference and roll-back operations, i.e. intermediate processing
stages). Technical infrastructures which support the resource management
are available (cf. e.g. Laudatio project12 or DSpace open source repository
application13 ) and start to get adopted on a larger scale for example in thecontext of CLARIN14 .
11https://www.go-fair.org/fair-principles/
http://www.laudatio-repository.org/laudatio
13
http://www.dspace.org/
14
https://lindat.m.cuni.cz/en
12

36

Chapter 2. Corpus Linguistics

2.4.2 Procedures in corpus analysis
Procedures for corpus analysis provide guidance for how linguistic insightcan be derived from empirical corpus data in a systematic fashion, basedon qualitative, quantitative and contextual information. The literature discusses dierent overall approaches to corpus analysis, and describes concrete workows and analysis methods.

2.4.2.1 Analysis approaches
As described in Section 2.2 and Figure 2.2, historically two approaches
to corpus analysis have been distinguished: hypothesis-driven analyses(corpus-based ) and data inductive analyses (corpus-driven ).
Hypothesis-driven analyses take corpora as empirical language basis
for validating linguistic theories. Theoretical assumptions are transformed
into concrete research hypotheses which are then tested on corpus data.15
Hypothesis-based analyses are the type of corpus investigations that are mostdocumented in the scientic literature (cf. Egbert et al. (2020, p.39)).
Data inductive analyses approach corpus data in search of new linguistic insights or inspiration. Inductive analyses build insights from data ina bottom-up fashion (cf. e.g. Tognini-Bonelli (2001)) instead of validating
existing linguistic theory. Open-ended corpus analyses are rather scarcely
discussed in the corpus linguistics literature (cf. e.g. Wilkinson (2007)), butthere is some indication that in practice open-ended analyses often blend
into the more common (i.e. hypothesis-driven) analysis workows (cf. e.g.
Gilquin and Gries (2009)).

2.4.2.2 Analysis steps
Corpus analysis workows fundamentally consist in three steps: the extraction of relevant data, their processing and analysis, and the interpretation and description of results. How the analysis steps are carried out
can dier between the dierent approaches to analysis.
The extraction of relevant data from a corpus can be more or less targeted. In hypothesis-driven studies it is well dened which data is neededin order to validate the theory. For open-ended studies, to the contrary,
the target of interest is potentially less specied and might just relate toa general linguistic principle (such as e.g. word order ). Data extraction
is usually done by means of computational search tools. Query languages
allow for the directed retrieval of data based on the specication of textual
elements, annotation layers and metadata. Depending on the analysis approach and objective, the data extraction can be based on a ne-grained15

For example, McEnery and Wilson (2005, ch. 6, p. 165 .) discuss a corpus study
on a linguistic hypothesis on sublanguages by giving a detailed account of three dierent
investigations.

2.5. Data and tools

37

query specication for a linguistic structure or consist in a simple listing ofa coarse set of feature dimensions of interest for analysis.
The corpus processing and analysis can relate to qualitative, quantitative and contextual information of the corpus data. The processing andanalysis of data is widely carried out by means of computational tools.
Major methods of analysis include 'listing, sorting, counting and concordancing' (Kennedy, 1998, p. 204) as well as advanced statistical analyses. In particular, data is often inspected in context (concordances ) andis manually classied prior to detailed statistical evaluations. For example,
16 help to show common word sequences and collocations' and 'help'KWICsidentify and disambiguate dierent word senses and revealing their distribution/relative frequencies of the dierent uses' (Kennedy, 1998, ch.

4.2). In
relation to the approach of analysis it can result in sets of examples basedon categorical distinctions or aggregated information derived from thedata.
Quantitative methods are employed for monitoring regularities, comparing and distinguishing language features, and generally for dening anddescribing recurrent characteristics of a language. Statistical techniques
range from simple and normalized frequency counts, proportions and distributions, over tests of signicance (e.g. chi-square, t-test, etc.) to multivariate statistical analyses and mutual information scores (e.g. for collocation
analysis).
The interpretation and description of analysis results consists in relating ndings to the initial research question or hypothesis under study.
The interpretation step is the human reasoning about the data and related analyses derived from the corpus. As expressed by Hunston (2002, p.
20) [...] a corpus by itself can do nothing at all, being nothing more thana store of used language'. By interpreting the analyzed results the corpus
linguist gives empirically-grounded explanations of the 'why' and 'how' ofwhat has been found in the corpus data.
2.5

Data and tools

This section discusses the empirical research basis of corpus linguistics interms of corpora, and software tools for their processing. First, a denition of the term corpus is given in Section 2.5.1, and the dierent types,
characteristics, annotations and formats of corpora are detailed (cf. Sections
2.5.1.1 to 2.5.1.3). Second, an overview of available types of computer tools
for the creation, processing, search and analysis of corpora is presented in
Section 2.5.2. Finally, advancements in corpus tools up to today are
discussed in Section 2.5.3.
16

Key Word In Context displays

38

Chapter 2. Corpus Linguistics

2.5.1 Corpora
A corpus is 'a computerised collection of authentic texts, amenable to automatic or semi-automatic processing or analysis. The texts are selected according to explicit criteria in order to capture the regularities of a language,a language variety or a sub-language' (Tognini-Bonelli, 2001, p. 55).

2.5.1.1 Corpus types
Corpora are characterized along dierent data characteristics (cf. e.g.
Kennedy (1998, p. 60 .)):1. Language form: written, spoken, signed
2. Data medium: text, audio, multi-modal
3. Type of language: general language, specialized language, learner
language, computer mediated communication, web texts, etc.
4. Sub-language: according to register, level of formality, genre, domain
5. Language: English, German, Japanese, Icelandic, etc.
6. Language coverage: monolingual, bi- and multilingual (parallel vs.
comparable corpora )7. Temporal dimension: diachronic vs. monitor corpora
8. Degree of processing: raw texts vs. annotated texts
For bi- and multilingual corpora a distinction is made between parallel
and comparable corpora. Parallel corpora contain texts together with
their translations into one or several other languages (i.e. originals andtranslated copies in multiple languages). Comparable corpora containtexts in multiple languages which are no translations of one another, but
are comparable in terms of their content or style. For comparable corpora
also the overall composition of text collections has to be comparable acrosslanguages.
Diachronic corpora contain texts of a dened time span. Monitor
corpora also contain texts of a certain time period, but get continuously
updated and enhanced with newly emerging texts. They serve to monitor
current language change.
Reference corpora are general language corpora which are representative for a language (e.g., standard German), instead of a particular sublanguage (e.g. the language of adolescents). They serve as basis for generallanguage studies (e.g., analysis of the South Tyrolean variety of German) andas a point of reference for studies on specialized languages or sublanguages(e.g., language of German speaking adolescents in South Tyrol).

2.5. Data and tools

39

Web corpora constitute another particular corpus type. They are entirely made up of texts from the Web. Web corpora are typically large insize and opportunistic with respect to the text selection17 , since the texts areusually collected from the Web in an automated fashion (e.g. by crawling
tools, cf. Section 2.5.2).
2.5.1.2 Corpus annotations
Corpus annotations are any explicit descriptions attached to the raw corpus texts. They comprise annotations of the text structure or layout (e.g.
mark-up of headers, titles, and paragraphs, as well as sentence boundaries),
linguistic descriptions (e.g. word class information or syntactic relations),
and metadata (information about the corpus texts, such as year of publication, author, etc.). They serve to enable advanced automatic retrieval andanalysis of the corpus data.
Linguistic annotations range from morpho-syntactic and word-levelannotations (e.g. lemma information), over structural patterns (e.g. syntactic relations) up to semantic and discourse features, etc.
Metadata provide additional details about the corpus texts, such as information about its authors (e.g. name, mother tongue, place of birth, etc.),
context and purpose of its production (e.g. place and date of publication,text type, language, etc.), its content, etc. Also formal and technical details of the corpus texts are recorded as metadata (e.g. character encoding,unique identier, link to source medium, etc.).
The annotation of corpus data is carried out (semi-)automatically bymeans of text processing and annotation tools (see Section 2.5.2.1) or
manually with specialized editors (see Section 2.5.2.2).

2.5.1.3 Corpus storage and data formats
Corpora are processed and stored in data formats that allow for the encoding of annotations, the exchange of data, and for the searching by automatic
tools. Annotations, in particular multiple or overlapping annotations, canpose challenges for data storage, exchange and retrieval.
Dierent annotation standards and best practices for data storageand exchange have been proposed and get extended over time. For example,
data formats have evolved from vertical le formats adapted for the indexingof token-level annotations, over texts with inline annotations for multiple
attributes (cf. e.g. LOB style inline annotation18 ), to XML-based stand-o
annotations following dierent schemas (cf. e.g. MMAX2 (Müller, 2005)).
Also, eorts to support the seamless transformation between dierent
data formats or the exchange and use of dierent formats within one appli17
18

Collected according to availability other than on pre-dened sampling criteria.
urlhttp://clu.uni.no/icame/manuals/LOBMAN/LOB2.HTM

40

Chapter 2. Corpus Linguistics

cation are ongoing (cf. e.g. PAULA (Chiarcos et al., 2008) or SaltNPepper(Zipser and Romary, 2010)).

2.5.2 Corpus software tools
Corpus linguistics makes use of a range of computational tools, which
support specic aspects of the corpus linguistics' workow, including dataprocessing and storage, as well as search and analysis of corpus texts.
In particular, research and developments in Computational Linguistics (CL),
Human Language Technologies (HLT), and Natural Language Processing(NLP) have led to the existence of a wide range of tools with continuously
expanding language coverage and performance levels. Nowadays, multiple
tools for corpus analysis, and various linguistic annotation tools are available for a large set of European languages and other major languages of theworld.

2.5.2.1 Tools for corpus building
Tools for corpus building (cf. Section 2.4.1) include tools for data collection, digitization, conversion and cleaning, and linguistic text processing, as
well as advanced editing tools for the manual annotation of corpora.
Next to generic web crawling tools (cf. e.g. Apache Nutch19 , Scrapy20 ,etc.), more specialized tools exist for collecting web corpora by harvestingtexts from the web. For example Baroni and Bernardini (2004) implemented
the BootCaT toolkit, a suite of perl programs for bootstrapping specialized
corpora and terms from the web.
Tools for the digitization of text by means of Optical Character Recognition (OCR) are well advanced and provide commercial solutions for printed
text documents of dierent types (c.f. e.g. ABBYY FineReader21 ) as well
as research tools for the automatic recognition of handwritten text (cf. e.g.
22
Transcribus platform ).
Other tools and scripts deal with data cleaning, such as the removal of
html-boilerplate from web texts (cf. e.g. Schäfer (2016)), or encoding andformat conversions (cf. e.g. Zipser and Romary (2010) for an approachto the mapping of annotation formats).
Linguistic annotation tools allow for the automatic processing of language data on dierent levels of linguistic description. Depending on thecomplexity of the annotation task, automatic tools reach dierent performance levels. While tools for lower levels of linguistic description (such as
19

http://nutch.apache.org
https://scrapy.org/
21
https://www.abbyy.com/en-eu/nereader/22
https://transkribus.eu
20

2.5. Data and tools

41

e.g. lemmatization and word class tagging) can be suciently reliable forreal application purposes, other tools may require manual post-processing.
The most common and diused types of NLP tools include sentence
splitter, tokenizer, lemmatizer and stemmer, part-of-speech taggers
and morphological analyzers, and dierent types of syntactic parsers
and chunkers (cf. e.g. Jurafsky and Martin (2000); Manning and Schütze(1999)).

2.5.2.2 Tools for manual corpus processing and annotation
Tools for manual corpus processing and annotation provide computational environments for the digitization, editing and mark-up of corpus texts.
They serve tasks like the transcription of handwritten text or spoken
language data, the post-correction of OCRed text, the correctionof automatically assigned annotations or the manual annotation of
specic features of the language, which cannot be determined by automaticmeans.
They are adapted for managing digitized corpus texts together with theirprimary sources (e.g. scaned images of handwritten text or audio les), andfor applying annotations of various types (i.e. metadata and comments at
document level, annotations at token or phrase level, etc.).
Established tools for the manual transcription and annotation ofaudio and video data include ELAN 23 (Brugman and Russel, 2004), a tool
developed by The Language Archive (TLA)24 for multimodality research andwith support for sign language corpora, and EXmARALDA25 (Schmidt and
Wörner, 2009), which provides support for the computer assisted transcription and annotation of spoken language (including the association of text
and audio recordings). f4transkript 26 is another basic transcription tool forthe digitization of interview data from audio les.
Tools for manual linguistic annotations include MMAX2 (Müller andStrube, 2006), a text annotation tool for the preparation of multi-layer annotated corpora, based on a exible stand-o XML data format, and We27 (Yimam et al., 2013), a general purpose web-based annotationbAnno
tool for a wide range of linguistic annotations, which also oers annotation
project management and freely congurable tagsets. SALTO (Burchardt
et al., 2006) supports manual annotation of text corpora and annotation
management, with special focus on annotating (frame) semantic argument
structures. ATLAS.ti 28 (Friese, 2011) is a widely used tool for qualitative23

http://tla.mpi.nl/tools/tla-tools/elan/
Max Planck Institute for Psycholinguistics, TLA, Nijmegen, The Netherlands
25
www.exmaralda.org
26
http://www.audiotranskription.de/27
https://webanno.github.io/webanno/
28
http://atlasti.com/
24

42

Chapter 2. Corpus Linguistics

data analysis, which supports the management, arrangement and annotationof content aspects and metadata of text documents together with graphic
and audio data.
These and similar tools for manual corpus annotation typically comewith GUIs or web interfaces and oer functionalities not only for applying
annotations but also for querying and visualizing the annotated data.

2.5.2.3 Tools for corpus search and analysis
Tools for corpus search and analysis (cf. Section 2.4.2) or short corpus
analysis software handle all aspects of corpus management including data
organization and storage, retrieval and aggregation of data, as well as data
analysis and display. They usually combine
1. support for corpus querying to retrieve concordances and manipulate
their order, classication and display, with
2. support for producing aggregated results like word frequency lists,
keyword lists, lists of collocations or n-grams, but also distribution anddispersion information for search hits based on dierent kinds of statistical measures.
Indeed, in corpus analysis tools the following functionalities are considered
standard functionalities: concordancing, collocations, frequency lists, keywords, thinning, sorting, frequency distribution and dispersion (cf. e.g., Scott(2008); Hardie (2012); Machálek (2020).
More specically, corpus querying allows to search a corpus for userdened textual elements such as words, and for inspecting the retrieved
results (concordances) in their textual context as KWIC displays (KeyWord In Context ). The search targets are dened by means of more or lesssophisticated query languages (cf. e.g. CWB/CQL (Evert and Hardie,
2011) or ANNIS/AQL (Krause and Zeldes, 2016)), which typically allow to
combine element specications on dierent annotation layers (i.e. strings
corresponding to words, lemmas, semantic labels, etc.), and support regularexpressions and boolean operators. Text-based as well as graphical searches(cf. e.g. TigerSearch (König and Lezius, 2003)) have been implemented forquery tools (Lemnitzer and Zinsmeister, 2010, p. 93)). The tools returnconcordances (KWIC), which can be counted or sorted by dierent criteria (alphabetical order, frequency ) in relation to a reference word position. In
addition, query tools usually provide dierent options for the results display(adjustable context window, highlighting), and export/download.
Corpus analysis tools are available as stand-alone software packages2.5. Data and tools

43(e.g. WordSmith Tools29 , AntConc30 , CorpusExplorer31 , #LancsBox32 , etc.)
and client-server based query environments for installation (e.g. ANNIS33 , IMS Open Corpus Workbench34 , NoSketch Engine35 or ICARUS36 ,etc.). Furthermore, several integrated online services provide direct access to readily available corpora (e.g. Sketch Engine37 , WebCorp Linguists'
Search Engine38 , English-Corpora.org39 , BNCweb40 , CQPweb41 , KorAP42 ,DWDS Search43 , etc.), or even the World Wide Web itself (see WebCorp
live44 ). In addition, also general purpose text indexing platforms areused for corpus storage and searching (cf. e.g. Lucene-based Apache Solr45 ).
In addition to standard search and concordancing, also functionalities for
global statistical analyses are available, which allow to computationallyprocess a corpus with regard to a specic aspect or feature dimension. Global
analysis functionalities include term extraction and collocation analysis, as well as multidimensional analyses and clustering of dened text
features. They typically output aggregated results, such as lists of collocations or key terms or tables with overuse and underuse statistics regarding
specic text features, or patterns of linguistic structures. Collocation andkeyword extraction functionalities are typically integrated within standard
corpus analysis tools (e.g. keyword extraction in AntConc46 ), but can also
be found as stand-alone applications (cf. e.g., TermeX (Dela£ et al., 2009),
47
TAPoR , etc.). More specic functionalities can be unique to single tools,
for example, the SketchEngine tool provides WordSketches which are summary reports of recurrent linguistic structures around a linguistic element(e.g. a verb together with all its valid syntactic complements and their frequencies (Kilgarri, 2002)).

29

http://www.lexically.net/wordsmith/
http://www.laurenceanthony.net/software/antconc/
31
www.corpusexplorer.de32
http://corpora.lancs.ac.uk/lancsbox/
33
http://corpus-tools.org/annis/
34
http://cwb.sourceforge.net/
35
https://nlp..muni.cz/trac/noske36
http://www.ims.uni-stuttgart.de/forschung/ressourcen/werkzeuge/icarus.html
37
https://www.sketchengine.co.uk
38
http://wse1.webcorp.org.uk/
39
https://www.english-corpora.org/
40
http://bncweb.lancs.ac.uk/bncwebSignup
41
https://cqpweb.lancs.ac.uk/
42
https://korap.ids-mannheim.de/kalamar
43
https://www.dwds.de/44
http://www.webcorp.org.uk
45
http://lucene.apache.org/solr/
46
https://wmtang.org/2014/07/12/keywords-list-antconc/
47
http://tapor.ca
30

44

Chapter 2. Corpus Linguistics

2.5.3 Advancements of tools for corpus linguistics
The number, variety and technical maturity of relevant software for corpus linguistic tasks has steadily increased since the beginnings of corpus
linguistics.
Regarding tools for natural language processing (NLP), in particular the number and range of tools for dierent annotation tasks as wellas their performance levels and reliability have greatly increased over time.
McEnery and Hardie (2012, p.31) report that nowadays for English andother major languages next to lemmatisers or morphological stemmers andpart-of-speech taggers, also constituency and dependency parsers, as well
as semantic taggers are in use for automatic text analysis. While support
for many other languages is still much more limited, overall the coverageof tools for various languages and scripts has steadily been extended, andrecent achievements of deep learning methods promise to lead to further
advancements in the near future.
With regards to tools for corpus analysis, McEnery and Hardie (2012,
p. 37 .) summarize major developments along four generations of tools.
The rst generation of corpus analysis tools were running on mainframe
computers at single sites. They supported only ASCII text and annotations were handled as idiosyncratic encodings within the text.48 However,already at these early times the tools allowed to create concordances andfrequency lists. The second generation of tools arose with the spread of
personal computers, and were adapted to run on several PCs. In termsof functionality they did not advance over their predecessors, and due to reduced processing power the size of corpora that could be processed with the
new generation of tools even decreased49 . With the third generation oftools (including WordSmith Tools (Scott, 1996, 2008), AntConc (Anthony,
2013a), MonoConc Pro (Barlow, 2000), etc.) the level of maturity that weare used to today was reached. Third generation tools are ready-to-use
desktop programs which combine various standard functionalities in one
application, they are capable to handle sizeable corpora with or without annotations, and they provide support for multiple languages and encodings.
The main innovation introduced by the fourth generation of tools is theirclient-server based setup, which provides access to the corpus search via
a browser. The most widely known and used examples are the corpus interface at english-corpora.org50 , the SketchEngine (Kilgarri et al., 2004, 2014),
and the CQPweb (Hardie, 2012) and BNCweb (Homann and Evert, 2006)
corpus analysis systems built on the IMS Open Corpus Workbench (Christ,
1994) and the MySQL relational database.
48

For example, in the LOB corpus part-of-speech tags were directly attached to theword by an underscore, e.g. dog_NN.
49
From a million words on mainframe computers to several 10,000 words on a PC.
50
https://www.english-corpora.org/

2.5. Data and tools

45

The advancements from one generation of tools to the next mainly relate to technical improvements, while on the functional level corpus tools
continue to mainly rely on standard functionalities like concordancing andlisting of words and their occurrence frequencies. Improvements with regard to technical aspects are related to the speed and scalability, support
for dierent languages and character encodings, and the portability of tools
across operating systems. Also, the development of ever more powerful
query languages and support for querying corpora with complex annotation layers has continuously advanced. In terms of new functionalities
and features that go beyond the standard solutions (see Section 2.5.2.3)
little innovation could be observed for many years, as Anthony (2013b, p.
151) attests '[i]nterestingly, the majority of the tool concepts proposed in the
1960s still serve as the foundations of modern corpus tools', and he speciesmore closely (Anthony, 2013b, p. 154):
In general, the current tools available to corpus linguists are fast
and feature rich.

They also oer researchers access to a wide

range of functions to analyze KWIC concordancers, distribution
plots, clusters and N-grams, collocates, word frequencies, andkeywords.

Indeed, still up to today advancements on the functional level aremainly found in relation to two aspects: the integration of more advanced
statistical tools (cf. e.g., Hardie (2012); Anthony (2013a); de Does et al.
(2017); Brezina et al. (2020)) and the visualization of search results (cf.
e.g., Kocincová et al. (2015); Rayson et al. (2016); Krause and Zeldes (2016);
Rüdiger (2018); Brezina et al. (2020), and Section 7.6.3)
More recently, eorts for advancing the transparency of corpus content, automated processes within tools (e.g., ranking algorithms) and analysis paths of users (e.g., query history and parameters, documentation of
resources, tools and queries) are increasing (cf. e.g. Kilgarri et al. (2014);
Hardie (2012); de Does et al. (2017); Machálek (2020); Brezina and team(2020), which perfectly responds to the demands for transparency expressed
recently by Egbert et al. (2020). For example, Machálek (2020) proposes
advanced features for corpus inspection and subcorpus building as well as
for the storage and reuse of detailed query histories. Brezina and team(2020) present the Wizard tool in #LancsBox, which allows to automatically compile research reports composed of information about the resources,
information about the tools and bulk outputs for single or multiple search
queries.
Another major change in corpus tool developments can be observed on
the architectural level, which responds to the need for more exibility anda better usability for non-expert corpus users, and corpus use in language
teaching. For both, analysis and annotation tools, the general development

46

Chapter 2. Corpus Linguistics

trend goes towards the creation of modular and exible corpus architectures, which are usually client-server based and increasingly support theuse of plug-ins, or as de Does et al. (2017) puts it:[T]he aim is not to develop a monolithic application that satises all requirements, but rather the development of a platformthat supports quick development of the analysis scripts and userinterface elements that are necessary for a research use case.

Relevant related services include for example ANNIS51 (Krause and Zeldes,
2016), BlackLab52 (de Does et al., 2017), CQPweb53 (Hardie, 2012), CorpusExplorer54 (Rüdiger, 2018), KorAP55 (Diewald et al., 2016), WebAnno56(Eckart de Castilho et al., 2014), and Language Resource Switchboard57 .
Table 2.2 provides a summary of new features and future works reported
in recent publications on a number of corpus analysis environments. As discussed above, advancements are still limited and mainly relate to the overallarchitecture of the tools, and its technical and usability aspects, with anincreasing number of more recent eorts to expand on statistics and visualization functions.

51

http://corpus-tools.org/annis/
https://inl.github.io/BlackLab/
53
https://cqpweb.lancs.ac.uk/
54
https://notes.jan-oliver-ruediger.de/software/corpusexplorer-overview/55
https://github.com/KorAP56
https://webanno.github.io/webanno/
57
http://weblicht.sfs.uni-tuebingen.de/clrs
52

Tool

ANNIS3(Krauseand Zeldes,
2016)
AntConc(Anthony,
2013a)
BlackLab
searchengine(deDoes
et al., 2017)
CQPweb(Hardie,2012)

Aim for updatingarchitectureforgeneric corpus searchand visualization
increase speed, support drag and drop,
annotatedles,statistics support
facilitate
maintenance and improve
technical aspects
increase usability andpower by adding exibility to use your own
corporaNew featuresgeneral graph-based data-structure (model)
with multiple segmentation layers, congurable
annotation triggered style sheets for visualization
include statistical modules via Python, more
modern look and feel, color highlighting ofsearch results, advanced search on large and annotated corpora, customizable word denitions
modular and exible, scalable, incremental indexing, separation of front-end and back-end,
open source, support for CQL, development inJavasimplied query language, history of query renements, saving queries, client-server modular
architecture

Future work

API access to allow document visualization

2.5. Data and tools

Table 2.2: Documented advancements in corpus analysis software

support for larger corpora, fast complex searches, distributed and multi-corpus
search,(across dierent
resources), subcorpus support, advanced
statistics and visualization support
implement analytic techniques as userfriendly functionalities, e.g.
(Biber, 1988),(Phillips,
1989),(Stefanowitsch and Gries, 2003)
Continued on next page

'chaining search''multidimensional' analysis'collocational networks' technique'collostructional' analysis

47

Table 2.2 Documented advancements in corpus analysis software  continued from previous pageAim for updating
New features
Future work

Kontext(Machálek,2020)

advanced,highlycustomizable corpus
query interface as
open-source projectwith improved userinterface and integration of search service
with other language
resources (LRs)support for recognizing statistical signicance of results, support for UD tokenized
and tagged texts, visualization of timebased data

API for word sketches, GDEX function, bilingual sketches, statistical corpus comparison,term extraction, extend localization, usability:
to log path to a search result, display more/less data for word sketches,thesaurus word cloud

SketchEngine's research agenda: paralleland distributed processing for large corpora, building large corpora from the web,, extrinsic
evaluatin and comparison of corpora, diachronic corpus analysis

based'

tion''proportion-based''inspection-'token-connect''kwic-connect''2-dimensional frequency distribu-'breadcrumb trail'

extend support for dierent languages'getting to know your corpus'

Chapter 2. Corpus Linguistics#LancsBox
support new ways of(Brezinasummarizing, analyzand team, ing and visualizing
2020; Brezina et al.,2020)
SketchEngine(Kilgarri
et al., 2014)

support for query construction, advanced query
history (i.e., including search parameters for reproducibility), support corpus examination andsubcorpus building (criteria-based,and, improved resultspresentation (i.e., including logs of manipulation operations), look-up of external LRs(via
andplugins, plug-in architecture, frequency interrelations (i.e.,), performance optimization
synthesis through dierent statistics tools, visualization of collocations, Wizard tool to automatically create research reports

48

Tool

2.6. User and usability research in CL
2.6

49

User and usability research in CL

Studies that investigate on the use of corpora and corpus softwarefrom the perspective of their users are rather sparse, in particular in
relation to linguistic usage scenarios. This concerns usability studies anduser-centered software design as much as systematic studies on the users'
objectives, behavior and needs towards corpora on a more general level. An
exception is found in the area of corpus use for language teaching, whereconcern with studying corpus usage has taken up speed in the last 10 to 15
years and numerous user studies are available by now (Boulton, 2013).
Related works on user and usability research are presented in three groups
distinguishing (1) user-centered studies on corpus usage situations(Section 2.6.1), (2) system reviews and user feedback on existing corpus tools (Section 2.6.2), and (3) requirements analyses for informed
software design (Section 2.6.3).
Furthermore, user-centered research in other domains concerned withthe discovery of insight and knowledge from textual data and related
methodological challenges are discussed in Chapter 3, Section 3.3.

2.6.1 Studies on corpus usage
Empirical research on corpus usage is found mainly in the context oflanguage and translation teaching. Small scale studies carried out
with a limited number of students have been reported since early on (cf. e.g.,
St. John (2001); Kennedy and Miceli (2001); Yoon (2005); FrankenbergGarcia (2005); Boulton (2012)). These and similar investigations largely
focus on evaluating the overall use of corpora for language teaching, where
corpora are mainly understood as concordances or word and collocation lists.
They investigate if corpus work benets the students learning eorts and how
students cope with the methods and tools for using corpora. The language
teaching contexts that are studied span various scenarios, including beginners
and advanced levels, as well as academic writing or translation courses. Over
time various small-scale user studies have been published, of which Boulton(2013) presents a corpus-based meta review. By extracting the mostfrequent lexical items from a corpus of 110 related English research papers
dating from 1989 to 2012 he constructs a short text 'summarizing' the content
of the collection of research papers as follows (Boulton, 2013, p. 53):
A group of learners, generally students, using data from texts
based on corpora for learning language or writing through concordancing. According to their level, they can look at both vocabulary (words) or grammar to gain information from the examplesof actual use given in a concordance, and search for patterns toimprove their knowledge in their English course. This teaching

50

Chapter 2. Corpus Linguistics
approachis known as DDL, and each research study provides

analysis of results from a test.

As this summary indicates, the studies generally attest that the lookup
of examples of actual language use in context benets language learning,
while the studies dierentiate to what extent corpus functions are helpful
for dierent learning settings (cf. Boulton (2012) and Frankenberg-Garcia(2005)). Boulton (2012) for example found that learners are in the rst placeinterested in nding examples that attest certain structures of languageuse (as found in concordances) and refer to frequency information rather
in a second step to choose between dierent options. In addition, studies
are also concerned with understanding how students should be introduced
to the corpus methodology such that they can make best use of it (cf. e.g.
Kennedy and Miceli (2001)), that is the pedagogy for corpus use inlanguage teaching.
Another larger-scale investigation has been carried out by Tribble (2012),
who launched an online survey on the use of corpus tools in language
teaching based on which was reported 'that the free online corpus interface
corpus.byu.edu, followed by the AntConc and WordSmith Tools for personal
computers, are the by far most used tools for analyzing corpora' (Anthony,
2013b).58
As opposed to the aims of this thesis, the user studies in language andtranslation teaching aim at evaluating the general use of corpus work
for language learning and of the currently available methods and tools
for using corpora. They are less targeted to identifying individual analysis
aims of their users or requirements for specic tool functionalities.

2.6.2 System reviews and feedback reports from users
In the past, several reviews of concordancing software have been published (cf. Reppen (2001); Diniz (2005)). Among the most recent (though
not so recent) and most comprehensive ones, Wiechmann and Fuhs (2006)
present a comparative review of 10 concordancers. The software tools
are reviewed along a detailed list of features, distinguishing aspects related
to functionality, performance and usability. For each tool the availabilityof all features is checked and a general evaluation of the pros and cons ofeach software are given. Similarly, Ari (2006) provides a review of software
to identify lexical bundles. He reviews the software tools for eciency
and user-friendliness, where eciency is dened as the capability 'to fullcriteria by which word combinations qualify as lexical bundles' and userfriendliness is understood as the 'ease in operating the interface also forpeople with little experience in using computers'. Also, Anagnostou and Weir
58

Unfortunately results were only published through a speech but not in written.

2.6. User and usability research in CL

51(2006) compare software applications for deriving collocations, basedon a list of functional properties relevant for collocation research.
Another type of user evaluation comes from constant feedback on existing tools that tool developers receive from their customers. For example,
de Does et al. (2017) report on a list of requirements that derive from useof the Blacklab search engine in dierent teaching sessions and research
projects. The requirements relate to powerful and parameterized query options, advanced grouping, sorting and categorizing of results, query history
and export of results, and comparing frequencies and distribution of results
to multiple queries (de Does et al., 2017, p. 252/253). The authors sum
up the wish list of major future developments, which includes a distributed
search to make searches faster and keep larger datasets manageable, searches
for tree-and graph-like structures, 'Chaining-Search' a serial combination of
searches in heterogeneous datasets and extended support for statistics, visualizations and subcorpus management. Finally they stress, that the overall
aim 'is not to develop a monolithic application that satises all requirements,
but rather the development of a platform that supports quick development of
the analysis scripts and user interface elements that are necessary for a research use case' (p.

256). A view on corpus software developments that is
becoming a common trend in the community (cf. Section 2.5.3).
As opposed to the aims of this thesis, the systematic system evaluations
focus on comparing the range of available features and evaluating
their power and speed, instead of investigating about relevant new features. The feedback from users on existing systems provides insights into
additional requirements, which however are linked to the experiences withthe existing systems.

2.6.3 Requirements analyses for informed software design
A number of studies that investigate on user requirements for corpus
tools on a smaller scale are presented here. The reported studies are mostly
part of design and development eorts of corpus tools.
As one of the rst user-centered studies, Santos and Frankenberg-Garcia(2007) investigated on user interactions with the web-based interface
to the COMPARA59 corpus, a Portuguese English parallel corpus developed at Linguateca. The study aimed at evaluating the usability of theinterface to be able to improve the tool throughout further developments.
The investigation of user actions is based on the inspection of a combination of server and system-specic log-les from a period of more than 4 years
amounting to about 74,000 queries. Overall they gathered information about
origin and frequency of queries, preferences concerning interface options (inparticular Portuguese vs. English interface language and search direction)
59

https://www.linguateca.pt/COMPARA/

52

Chapter 2. Corpus Linguistics

and query types (simple vs. complex queries). They report that more than
97% of the complex queries request concordances as opposed to details about
the distribution of results. The authors conclude that their study permitted
them to 'detect some general usability problems as well as gather material
for further research and development of COMPARA' (p. 364). Usability
problems included empty search results due to wrong query syntax, wrong
language choice or misspellings. Improvements to the interface that could
immediately be implemented included for example improvements of the documentation, and simplication of case-insensitive queries or queries without
the use of diacritics. However, the authors also stress that the investigation'just scratched the surface of all there is to do in a serious user analysis of
a computational service' (p. 364).

Soehn et al. (2008) conducted a study on requirements for a userfriendly web-based corpus query interface as part of an initiative thataims at fostering sustainability of corpus resources by providing a single interface that gives access to a number of heterogeneous corpora. Design decisions
for the new interface are guided by insights from two studies: a requirement
analysis of user needs based on an open-ended questionnaire and aqualitative analysis of three existing corpus query interfaces. The
questionnaire that was compiled by 20 subjects from dierent areas of expertise (ranging from computational linguistics to linguistics/language studies
of dierent languages) distinguished three dierent aspects of query tools:
search, visualization and export of query results. The answers to the survey indicated what known functionalities are particularly important to theusers but did not yield requirements concerning innovative features. In thesame way the analysis of existing tools was used to determine what popularfeatures to adopt for the new interface, such as powerful query options that
support Unicode and the search on dierent annotation layers, the exible
display of selected annotation layers, and support for storing queries andquery results, etc.
More recently, Knight et al. (2020) collected user requirements fora corpus interface in the context of developing a computational infrastructure for The National Corpus of Contemporary Welsh (CorCenCC). Bymeans of a short survey corpus users were asked about their expertisewith tools and preferences towards functionalities ('Which functionalities do you use most commonly?', 'What functionalities would you liketo see added to your favorite tool(s)?' ).

The responses of 62 participants
indicated that keywords in context (KWIC), concordancing, frequency lists,
collocation analysis, visualization, keyword analysis and n-gram analysis are
the most required functionalities (in order of importance). In addition, theusability and functionality of the web-based interface was evaluatedby means of road testing with three experienced corpus linguists. They
were asked to freely explore the interface functionalities for one hour. During
the testing verbal feedback was collected through think aloud protocols and2.6. User and usability research in CL53

the activities of the testers were screen-captured. The testing, together witha post-test interview with each expert, yielded the following results. Overall,
the systems was perceived as relevant and easy to understand. Some technical problems were encountered and especially long processing times without
status information were confusing. Requests for additional functionalities
and features included support for Chi squared test, a translating feature,and the possibility to download search results. Also, more explanations on
the semantic of tags and the system processes underlying information likethe ranking score were desired.
Finally, as part of her PhD thesis, titled 'Corpus Linguistics Software:
understanding their usages and delivering two new tools', (Gomide Rodrigues,2020) carried out a literature analysis of 5,000 academic articles reporting corpus-based studies (dating from 2015 to 2017). The articles
were extracted from two academic databases and were analyzed with corpusbased methods. The analysis served to identify which corpus tools wereused for language research, how they were used and which features go
beyond concordancing and how the tools use distributes over dierent subdisciplines. Gomide Rodrigues could attest that 21 corpus softwaretools were mentioned, with BYU corpora60 , WordSmith Tools and AntConc
being the most used tools. As for functionalities that go beyond concordancing in particular the following ones were most used and desired: simple andsophisticated querying options including standardization of spelling variants,
visualizations for quantitative corpus data,61 and analyses and subcorpus
building based on metadata. As with regards to tools complementing corpus software, tools for statistical analysis, and again visualization were themost cited tools. In addition to the literature review, Gomide Rodrigues(2020) performed a small quantitative analysis using the methods of contextual enquiry to collect user requirements from ve corpus users with varying
proles. The quantitative study combined think aloud protocols and screen
casts of the users performing corpus analysis tasks of their choice. Results ofthe study were combined into an anity diagram with 45 requirement statements related to usability, processing, corpus management and functional
aspects among others (Gomide Rodrigues, 2020, p. 95). They mostly relate
to usability aspects and detailed demands related to existing functionalities,
such as 'oer intuitive exploration of statistical options' and 'display only
positive keywords by default'. In relation to metadata and visualization also
more wide-reaching requirements were stated, such as 'allow for comparisonof corpus parts' and 'display a visual summary of the data'.
As opposed to the aims of this thesis, the requirements analyses of related works present a combination of the evaluation of existing system60

now https://www.english-corpora.org/
It is noteworthy that tools that use visualization were only named by their creators(i.e., in articles introducing the tools), but did not appear in the description of any researchstudy.
61

54

Chapter 2. Corpus Linguistics

functionalities and small user investigations on very open-ended
questions (e.g., 'What functionalities would you like to see added to yourfavorite tool(s)?' as explained above), while this thesis aims to explore ina systematic fashion, what approaches are followed by users to make use of
corpora and how these procedures could be better supported by computational software tools. Out of the presented related works, the PhD study ofGomide Rodrigues (2020) relates most closely the objectives of this thesis inthat it shares the objective of reducing the gap between what corpus software tools are currently oering and the needs of their users. However, unlike
this thesis, it explicitly focuses on usability aspects for non-specialized
users rather than on functional innovations for specialized corpus
linguists. Also, the requirements analysis is linked to existing tools and issmaller in scale in terms of the number of users involved.
2.7

Conclusions and aims of this thesis

This chapter provided a detailed overview on the theoretical foundationsof corpus linguistics and the state of the art with regards to its overall
objectives, principles, application areas, methodologies and resources andcomputational tools.
The chapter started by describing how 'modern corpus linguistics'
arose, what theoretical and technical foundations it builds on and how it
evolved from the late 1950s up to today (Section 2.1). Section 2.2 gave adenition of corpus linguistics and laid out its major theoretical principles. In Section 2.3 a selection of its main application areas, as foundtoday, got exemplied. Section 2.4 gave an account of the methodologies
employed in corpus linguistics and detailed common procedures for corpus building and analysis. In Section 2.5 both the corpus resources
and computational tools for their processing were described. Information
on dierent corpus types as well as their annotations and storage formats wasgiven, and an overview of software tools for corpus preparation and analysis
was presented. Also, advancements of corpus tools since the beginnings ofcorpus linguistics were discussed. Finally, an overview of related works onuser-centered research in corpus linguistics was presented in Section 2.6.
The description provided in this chapter shows that by now corpus linguistics has become a well-established research eld that builds on different theoretical principles and methodological lines of thought (cf. Section
2.1 and 2.2), encompasses various application areas (cf. Section 2.3) andmakes use of a wide set of methods and tools (cf. Section 2.4 and 2.5). It
could also be observed that the range of functionalities oered by corpus
tools has evolved very moderately since the foundations of corpus linguistics(cf. 2.5.3). At the same time, a literature review has shown that usercentered approaches to software design and user studies in corpus linguistics

2.7. Conclusions and aims of this thesis55

are rather scarce (cf. Section 2.6).
Relating the state of the art in corpus linguistics to the research objectives of this thesis, we can observe that the theoretical groundings ofcorpus linguistics are well established and basic research procedures andtools are in place. We can also notice that less documentation and knowledge is available on the daily practice of corpus-based research and theconcrete experiences and demands of corpus users.
Given the expectation that application contexts for corpus linguistics willcontinue expanding and that the need for new approaches and methods for
corpus analysis will continue to evolve (cf. Section 2.4), we believe that weneed to focus more closely on the activities and needs of corpus users.
This is relevant both to formalize and further develop methodologies
for corpus analyses, and to implement tool functionalities that are able
to respond to specic user needs that go beyond the standard functionalities
available in corpus software tools since long (cf. Section 2.5.2.3).
This need for systematic insights into the users' needs is addressed by thisthesis by means of a user-centered study (see Chapter 4 and 5). It aims tocomplement the theoretical knowledge and established practical proceduresof the discipline with a user and usage-based perspective, in order togain a wholistic view on the users' experiences and needs in corpus research
work with regards to their overall motivation, methodological principles,
research objectives and practical and pragmatic realities. The user study
will be approached by means of a detailed enquiry of corpus users toanswer the following research question:

What are open requirements in current corpus research
practice and for related tools support?
In order to address the research question user investigations will enquiry
about six relevant sub-aspects as targeted by the following questions:
1. Who are the corpus users?
2. What are the general objectives of the corpus linguistics approach?
3. What are concrete research objectives tackled by means of corpus linguistics?
4. What are established methodological principles in corpus linguistics?
5. How is the methodology implemented by means of practical workows,
methods and tools?
6. What are requirements towards methodology and technological support?

56

Chapter 2. Corpus Linguistics

In this way, the study aims at documenting the corpus linguistics research
scenario from the users' perspective, at detecting open needs, and at deriving
related requirements towards computational support in software tools for
corpus users. The study focuses on corpus users that carry out linguistic
studies based on corpora and excludes any other type of corpus users (suchas e.g. corpus use in language teaching, translation studies etc.).
The overall approach of the user study is outlined in Section 3.4 and thedetailed user studies are presented in Chapter 4 and 5.

-Chapter3-

User-centered research

57

58

Chapter 3. User-centered design and evaluation

3.1. Introduction to human-centered research59

Aim of this chapter

This chapter aims at introducing human-centered approaches to research investigations and software design and development. It provides themethodological background for the user studies reported in Chapter 4
and 5. On the one hand, it introduces classic enquiry methods fromthe social sciences, which focus on investigating behaviours and attitudes ofhumans. On the other hand, it describes methods of human-centeredsoftware design, development and evaluation, which aim at improving software usability. In this chapter, we are taking a wide perspective on
the subject to prepare the ground for devising the user investigations of thisthesis which address corpus linguistics research aims and methodologies ingeneral and in relation to the use of computational tools in particular. Furthermore, we are pointing out challenges for human-centered usability
research in corpus linguistics by highlighting experiences from dierent
application domains that share characteristics with the corpus linguistics use
case.
The chapter starts by introducing objectives and approaches of humancentered research in general and the usability engineering approach in particular (Section 3.1). After that common methods of enquiry, including
data collection and analysis, are presented with a focus on the methods ofparticular relevance for the user studies of this thesis, namely interviews
and questionnaires (Section 3.2). Finally, general challenges in evaluating tools for open-ended research, analysis and information retrieval tasks
are discussed based on relevant examples from other domains (Section 3.3).
The chapter closes with a summary of the challenges for user-centeredresearch in corpus linguistics and the research approach of this thesis in
Section 3.4.
3.1

Introduction to human-centered research

Human-centered research is concerned with studying the attitudes,
behavior and (inter)actions of humans related to a specic situation,subject, approach or tool. The human-centered approach aims at understanding and describing a situation in order to derive requirements related to it and devise solutions to act on them. In doing so, it builds on
the assumption that, for evaluating and advancing subjects that humans
are involved with, it is of fundamental importance to understand the situation and context people are in and to take into account their opinions andperspectives.
Related research objectives can target theoretical insights on an abstract
subject or practical insights related to a physical or virtual object. Most

60

Chapter 3. User-centered design and evaluationquestions in social science research are concerned with theoretical insights
which can be drawn from studies of the behavior and interaction of humans.
To the contrary, any kind of design questions, including software design,
target practical insights regarding an object, tool or service that a person isinteracting with. Respectively, the research focus is on a person in its
role as 'user' of something, and related research is termed 'user-centered'research.
More specically regarding software design, Stone et al. (2005, p. 628)dene the user-centered approach as 'an approach to user interface designand development that views the knowledge about intended users of a system
as a central concern, including, for example, knowledge about user's abilities
and needs, their task(s), and the environment(s) within which they work'.

Overall, thus user-centered research aims at creating the necessary conditions
for adapting systems and services to users, instead of requiring users to adapt
to the tools they are using.
Human-centered or user-centered research is implemented by methods
and techniques for observing, investigating and analyzing the users and theircontextual realities. In addition to research in the Social Sciences, whichhave a wide variety of established methodologies for user investigations andeld studies, user-centered methodologies are developed and applied in market studies, ergonomics and (software) design.
The discipline of Human Computer Interaction (HCI) is particularly concerned with human-centered issues related to software design anddevelopment. It aims 'to improve the interaction between users and computers' (Kerren et al., 2007, p. ) and in order to do so it combines approaches
from several elds, including human factors, ergonomics, psychology, cognitive science, articial intelligence and computer science (cf. Leventhal andBarnes (2008, p. 15/16)).

3.1.1 The concept of 'usability'
The concept of usability is dened by the International Standards Organisation (ISO-9241-11, 1998) as 'the eectiveness, eciency and satisfaction
with which specied users can achieve specied goals in particular environments'.

Accordingly, any evaluation of usability is context dependent, withregard to characteristics of the users, their goals and the environment
in which a tool is employed (see below). Nielsen (1994) further included
learnability, memorability (ease of remember) and error tolerance among theparameters for evaluating usability.
Although all these aspects are relevant for the evaluation of usability,
some of the factors can be mutually dependent in opposing ways. For example, when comparing generic vs. specialized tools, it is likely to observe
a contrary balancing between learnability or ease of use and eectiveness ofthe tool.

3.1. Introduction to human-centered research

61

For the aspect user satisfaction more recently also the term 'user experience' has been established, and is primarily referring to the users' feelingsin relation to a system. Hassenzahl (2008) denes user experience (UX) as'a momentary, primarily evaluative feeling (good-bad) while interacting witha product or service.

Good UX is the consequence of fullling the humanneeds for autonomy, competence, stimulation (self-oriented) through inter-acting with the product or service (i.e. hedonic quality)'. An evaluation of
the user experience can stem both from pragmatic and hedonic aspects ofthe system. Despite its more emotionally anchored focus, user experience is
occasionally used interchangeable with usability.
The concepts users, goals and environment are briey dened in thefollowing subsections.

3.1.1.1 Users
A user is a person, that is employing a tool, technique, product or service
in order to achieve a specic objective. According to their characteristics,
users can be associated with dierent user types which are formalized into
user proles or user models (cf. e.g. Kulyk et al. (2006, p.39 .)).
Distinctions between user proles can relate to physical, cognitive andcultural characteristics of users, as well as to their competences and skills. In
particular, the following categories have been proposed for describing users(cf. Nielsen (1994), Kulyk et al. (2006, p. 37), Russell-Rose and Tate (2013,p. 4)):
 physical abilities and disabilities
 cognitive abilities
 personality characteristics
 cultural and international diversity
 degree of expertise

Regarding the users' expertise, Russell-Rose and Tate (2013, p. 4) pointout that domain expertise namely 'one's familiarity with a given subject
matter' should be distinguished from technical expertise, which is 'one's prociency at using computers, the Internet, search engines, and the like',as they have dierent implications regarding requirements for usability.

3.1.1.2 Goals
Goals are the overall objectives which a user aims to achieve. Consequently,
all user activities are driven by these goals, and thus understanding the users'
goals is fundamental for evaluating and improving the usability of tools.

62

Chapter 3. User-centered design and evaluation

The objectives of users can be investigated and described on dierent
levels of granularity. Hackos and Redish (1998, p. 51 .) introduce a distinction between goals, tasks and actions. User-centered investigations canaddress aims related to all three levels of objectives.
Goals describe the users' overall intentions, the nal outcomes users
wish to achieve. Goals can be very general such as 'to validate a linguistic
theory'. Understanding the goals of users is important, because '[...] focusing
on the high-level goal [...] will allow you to see possibilities for oering users

other ways of accomplishing their goals.' than they might have thought ofthemselves (Hackos and Redish, 1998, p. 255).
Tasks are the things 'someone does to achieve a goal' (Hackos and Redish, 1998), (p. 56). Tasks are more concrete than goals and are oftentimes
closer to the level of abstraction at which a user is conceptualizing his work,
e.g. a task related to the abovementioned goal might be 'to nd examples
that provide positive evidence for the theory' or more ne-grained 'to searcha corpus for e.g. passive verb constructions'. Usually, one specic goal canbe achieved by means of several dierent sets of tasks.
Actions are the concrete steps that the user has to undertake to accomplish a task. They can be as simple as 'to sit down at a computer', 'to open
a concordancing tool', or 'to type in a search word'. In the same way as withgoals and tasks also concerning tasks and actions users can choose between
several dierent actions for completing one and the same task.

3.1.1.3 Environment
The environment or context of use is the physical and relational usage situation in which a tool or method is employed. Next to characteristics of theuser and his/her goals, the environment is the third main aspect relevant tousability considerations.
The environment is characterized by concrete conditions of the workplace(including spatial situation, light, noise, single or mulitiple person oces,etc.), by the team working conguration (e.g. whether a person is working
on her tasks alone or in a team), and by factors related to timing (e.g. taskcompletion under time pressure, interruptions, etc.), etc. (cf. Jordan (1998)
Nielsen (1994)).

3.1.2 The usability engineering approach
Usability engineering is the 'structured approach to building software systems that meet the needs of users in various environments with varying levels(Good, 1988, p. 1). Usability engineering is situated at the crossroad between software engineering and HCI. Concerning its
status, Leventhal and Barnes (2008, p. 3) state that '[u]sability engineeringof computer experience'is now central to what many computer scientists and information technology

3.1. Introduction to human-centered researchprofessionals do'63

and '[e]valuating usability is now considered an essential

part of the system development process' (Dillon, 2001).

HCI provides process models and methods for usability engineering which
systematically involve the user at all processing stages. These software development procedures (cf. e.g. Hamborg and Gediga (2006)) typically startfrom requirements analysis and functionalities specication, followed by several cycles of prototyping, testing and improvements of the tools' functionalities. The development process is usually concluded with a nal evaluationof the developed system.
The dierent processing stages are briey introduced in the following
subsections. Further details and related methods for analysis and evaluation
can be found in Hamborg and Gediga (2006), Leventhal and Barnes (2008),
Jordan (1998), and Bortz and Döring (2006).

3.1.2.1 Requirements analysis
Starting point for the development of software with a high degree of usability is a requirements analysis. The aim of the requirements analysis is tounderstand the prole of the prospective user, her tasks and the environment
in which the software will be employed. It usually results in a requirements
specication that is taken as basis to determine demands towards the software, and serves to inform the system design. Requirements specications
contain detailed descriptions of tasks and workows and information about
tools or resources that are part of the usage context. Furthermore, the professional and social environment as well as characteristics of the potential
users and their variability are included into a requirements description.
For example, the requirements specication for an information system on
public transport could include the following requirements: optimization forthe display on portable devices (context of usage), option to store recurrent
travel routes, travel times (tasks), localization to various languages (user),storage of user prole (age, bonus cards, etc. of the user), etc.
Common methods for requirements analysis comprise interviews, questionnaires, focus groups, eld observations, contextual enquiry, and contextual task analysis.

3.1.2.2 Iterative prototyping
The iterative software development builds on results of the requirements
analysis. It aims at continuously improving the software product by successively reacting to updated requirements during the implementation process.
This approach for development is implemented by iterating development andevaluation phases: every new development is immediately tested for its usability, and possibly leads to adjustments of the requirements or it results in
new ones.

64

Chapter 3. User-centered design and evaluation

The development-evaluation cycles within iterative prototyping are relying on methods of user participation (including direct/indirect and active/passive participation), and on low and high delity prototypes1 , e.g.
screenshots, mock-ups, paper scenarios, animations, early system releases,
etc.

3.1.2.3 Evaluation
The evaluation phase aims to compare alternative solutions, to assess theusability of a specic system (or prototype) and to determine possible weak
points (Hamborg and Gediga, 2006, p. 11). Methods and criteria of theevaluation as well as specialized evaluation aims 'will usually be dependent
on the stage of the product creation process (PCP) at which the evaluation
will take place' (Jordan, 1998, p. 81).

Regarding available approaches to evaluation, a general distinction ismade between formative and summative evaluation. Formative evaluation isdone in the design phase and provides qualitative results, e.g. from individual
users' feedback. Summative evaluation is performed to evaluate a nalized
application and is based on quantitative results, e.g. of formalized tests ona larger scale (Kulyk et al., 2006, p. 59).
Several analytical and empirical methods have been proposed for all
evaluation stages. (Cognitive) walkthroughs, expert appraisals and group
design reviews are examples of analytical evaluation methods which providea systematic evaluation of system features by a usability expert (p. 56).
Eye-tracking experiments, analysis of log les, think-aloud protocols, video
recordings, interviews, questionnaires and usability tests belong to the groupof empirical evaluation methods, which 'collect usability data by observing or
measuring activities of end users interacting with a prototype or [...] system'(Kulyk et al., 2006, p. 57).
The setting of the evaluation poses another important distinction: controlled experiments (e.g. in laboratory settings) with designed analysis tasks
are accompanied by eld studies which evaluate how users solve potentially
open-ended authentic research tasks in their natural environments (p. 58).
Furthermore, longitudinal evaluations are distinct from evaluations carriedout at one specic point in time.
3.2

Methods of enquiry

A wide set of methods for user-centered enquiries, including methods fordata collection as well as for analysis of the collected data, has been pro1
The 'delity' of a prototype relates to its similarity with the intended nal product.
Accordingly, high delity prototypes are strongly approximating the nal product.

3.2. Methods of enquiry

65

posed within the social sciences, and also within human-computer interaction, particularly in connection with usability engineering approaches.
The following subsections describe methods for data collection (3.2.1)
and data analysis (3.2.2). Each subsection opens with a short overview on
the most widespread methods, and continues by describing in more detail themethods used within the user studies of this thesis (indicated as underlined).

3.2.1 Data collection methods
The descriptions of data collection methods are based on Bogner et al. (2005),
Bortz and Döring (2006), Hackos and Redish (1998), Helerich (2005), Iarossi(2006), Kulyk et al. (2006), Nielsen (1993), and Porst (2008).

Interviews are oral interactions between a researcher and an informant,
regarding a specic subject. Interviews can show dierent degrees ofstructuring based on the nature of the questions (open vs. closed) andthe level of guidance by the interviewer (predened set of questions vs.
few open-ended guiding questions).
Questionnaires are the written counterpart to interviews. They consist
in written compilations of a sequence of questions regarding a specic subject and serve to collect data from groups of informants in an
uniform way.
Focus groups are moderated discussion groups on a specic subject of interest, held between 6 to 12 informants. They serve to collect diversied information and views on a subject by recording semi-guided
discussions between persons with relevant knowledge and insight on
the subject.
Field observations are research visits in authentic real-life settings which
are recorded for analysis. They take the form of written protocols,
video or audio records and focus on a specic aspect of the setting
which relates to the research objective.
Contextual enquiry and participatory task observation are methods
for observing working procedures in real-life settings, possibly mixed
with targeted questions. They are carried out by a researcher witha user in an authentic user setting and are usually video or audiorecorded.
Expert protocols and expert reviews are short evaluations of softwaretools, along a set of dened evaluation parameters (e.g. heuristic evaluation), carried out by usability experts (or usage domain experts).

66

Chapter 3. User-centered design and evaluation

3.2.1.1 Interviews
Scientic interviews are a long-established survey method for collecting empirical data by means of asking questions in a conversational situation. Together with written surveys, interviews are the most applied enquiry methodin empirical social science and make up for about 90% of all data collection(cf. Bortz and Döring (2006)).
Historically, interviews have rst focussed on interrogating experts about
their domain knowledge. Subsequently they started to get employed for collecting opinions from dierent groups of people, which brought about new
challenges regarding methodologies and best practice guidelines. In fact, interviewing techniques aim at standardising the interviewing procedure andsituation in order to prevent bias introduced by the interviewer and interviewing situation and maximise objectivity, reliability and validity of thisresearch instrument. Nevertheless, the objectivity of information collectedby means of interviews, as well as questionnaire surveys, is inherently limitedby the fact that the data is based on the opinions and evaluations of the interviewees which can be biased and subjective, and will partly be hypothetical(Nielsen, 1993).
Compared to written surveys, interviews generally tend to elicit more
spontaneous and open, but less complete or comprehensive replies. Interviews allow for open-ended questions and adjustments to the ow of addressed topics (Nielsen, 1993), and thus provide a powerfull mean for the
exploratory enquiry of a topic.

Types of interviews
A wide variety of dierent interview types is described in the literature,
although denitions overlap and no denite typology has been established.
Primarily, interviews can be distinguished with reference to the degree ofstandardization, the interviewing situation, and the relation between
interviewee and interviewer.
The degree of standardization corresponds to the structuring andguidance of the interview and is related to the overall research objective,
i.e. whether the interview approaches an exploratory objective and aims at
a rst orientation and thematic structuring of a topic or whether it pursues
a systematizing objective and aims at harvesting information in a systematicand comprehensive fashion (Bogner et al., 2005). A higher degree of standardization results in a reduced openness regarding possible replies and theoverall interview ow, but also in less control regarding the adressed topics
and ow of discussion.
In fully structured and standardized interviews all questions are clearlyspecied, possibly down to the exact wording, and their sequential order ispredetermined and mandatory to follow. Questions are typically very spe-

3.2. Methods of enquiry67

cic and target short answers or provide standardised answer alternatives
that the interviewee can choose from. Structured interviews are particularly applicable for investigations on clearly contoured topics that build oncomprehensive existing knowledge.
Open interviews constitute the other end of the scale. They are basedon a thematic frame that sets the context for an open conversation, without
relying on predened questions. The interviewer is participating as attentive
listener that is only slightly guiding the conversation while the interviewee
is given room to freely talk and lead the ow of topics. Open interviews are
employed for gaining a rst orientation on new research areas, for carryingout exploratory investigations, and for investigating on delicate topics.
Semi-structured interviews provide for all intermediate degrees of standardization. A typical example are guided interviews that rely on a set ofloosely dened questions which provide a skeleton for the interview topics
but allow the respondent to expand on or limit particular aspects.
Regarding the interviewing situation, a distinction is made with respect to mode or location of the interaction, the number of interviewees andthe number of interviewers per interview.
Interviews can be carried out in direct contact (face-to-face), via telecommunication (e.g. phone or Skype), or in written form. Interviews via telecommunication are perceived as more anonymous. They tend to yield higher
participation rates, but lower response rates to sensitive questions2 . Teleinterviews are logistically easier to arrange and less time consuming.
Interviewing of individuals vs. groups of people typically induces dierent response dynamics. For example, interviewing of groups composed ofpeople on dierent hierarchical levels (e.g. superior and normal employee)
can increase the risk to encounter unnatural or biased answers.
On the side of the interviewer, the eect of single interviewers is to createa more condential atmosphere and be less intimidating. On the contrary,
having several interviewers can be benecial for dividing the conduction andrecording tasks and allows to be better equipped for grasping all information.
Regarding the relation between interviewee and interviewer, theinteriviewer's authority or the role she is assuming when turning to the interviewee has an impact on the interview. On the one hand, acting soft andunderstandable will help creating a comfortable interviewing situation andrise the interviewees condence. On the other hand, a hard and authoritarian
approach can be a strategy to force answers on uncomfortable topics. In addition, the interviewer's competences in the addressed domain are a relevant
factor. For example, an overly knowledgeable interviewer might intimidate
an interviewee, while to the contrary a highly competent interviewee could
be demotivated by an apparently unprepared or incompetent interviewer.
Despite these possibilities to inuence the interview by taking on particular2such as e.g. questions on drug consumption or income rates

68

Chapter 3. User-centered design and evaluation

roles, in most cases the interviewer would avoid to act in extreme roles, andwould rather aim at creating a balanced situation and express an objective
interest in information that the interviewee is able to provide.
The expert interview constitutes a special type of semi-structured interview, in which a domain expert is interrogated on his subject of expertise.
The expert knowledge can relate to factual information on a specic subject, process knowledge related to practical experience regarding a subject
or working context and interpretative knowledge, which reects the specicperspectives and conclusions of an expert on his subject of expertise (Bogner
et al., 2005, p. 43 .). In general terms, expert interviews put a strong focus
on the objective of retrieving content information from an interviewee while
widely disregarding the impact of the relation between the interviewer andinterviewee or the interviewing situation, which however cannot be prevented
completely. In fact, Bogner et al. (2005, p. 55 .) discuss common eects of
the interviewer's domain expertise, authority, and distance or familiarity on
the interview outcome.
In the context of usability engineering, Hackos and Redish (1998, p. 135),
suggest to use expert interviews for process analyses. In this prospective
users of a to-be-developed software are asked to systematically walk the interviewer through tasks and processes related to their work, while explaining
and commenting in detail on the steps involved. Employed in this way, expert interviews can serve to understand the user's working context, related
issues, and terminology. Starting a process analysis with a series of interviews implements a 'top-down' approach, while starting from e.g. contextual
enquiries would work 'bottom-up' (Hackos and Redish, 1998).

Design and order of interview questions
The questions in interviews serve a two-fold purpose: they are the instrument for retrieving information and for guiding the interview process.
Questions can target factual information, opinions and attitudes, andthey can address real or hypothetical subjects. Depending on the way questions are formulated they can trigger more or less open and elaborate answers. A general distinction can be made between open question, yes/noquestions and closed questions with predened answer categories.3
In their function as the steering instrument questions can be divided into
introductory questions, main questions, follow-up questions, bridging questions, and ltering questions (Gläser and Laudel, 2009, p. 130). The orderof questions can inuence the extent, detailedness and content of answers.
While answers tend to be inuenced by preceding questions buering questions can help to reduce these interference eects and preparatory or transi3
Closed questions are more applicable to questionnaires than to interviews. Thus
related considerations are discussed in Section 3.2.1.2 below.

3.2. Methods of enquiry

69tion questions can smoothen topic shifts. In addition, ltering questions canbe deployed to skip sections of the interview which are not applicable.
Regarding the overall structure of an interview, questions related to one
topic should be grouped together and the interview should follow the natural
chronology of questions, if applicable. An interview should not address morethan four major topic blocks.
Bortz and Döring (2006) (with reference to Bouchard (1976)), Gläser and
Laudel (2009), Helerich (2005), and Porst (2008) give the following detailed
guidelines concerning interview question design: reduce the number of questions as far as possible, avoid repetitions
 devise clear and easily understandable questions, with a unique infor-

mational aim (monolithic ) for adressing complex topics a sequence of several statements followed

by a short question can be used ('platform' question)

 avoid 'why' questions, as they tend to elicit unclear answers
 avoid double stimuli and negations
 ensure the correct and unambiguous use of concepts and vocabulary,dene unclear terms if necessary, use explicit temporal references devise questions which are neither too general nor too specic, avoid

abstract and categorizing language

 avoid problematic (e.g. intimidating) questions, in case phrase them as

hypothetical questions and put them towards the end of the interview

 in case polarised questions need to be used, they should be balanced

between positive and negative

 avoid presumptions and suggestive questions, check for implicit 'pre-suppositions' and whether they are acceptable or harmful/misleading

 make sure that the context does not inuence replies to a question
 take care that research questions are not directly asked but translated

into interview questions

 check if questions and informational needs match

70

Chapter 3. User-centered design and evaluation

Characteristics of the interviewer and interviewee
In interview-based research the interviewer acts as the retrieval tool andconstitutes a 'living research instrument'. Inevitably, on the behaviourable
level, the personality and overall appearance of the interviewer will have aneect on the process and outcome of the interview as it cannot be fully standardized. Attributes which characterize a professional interviewer include to
be attentive, neutral and self-reective, as well as controlled with regard to
expressing personal opinions or bias.
In addition, on the procedural and content level, the interviewer should
be familiar with the course of interview questions, she should be well informed about the context and conditions of the research study, and should
be knowledgable about the subject. She should also be informed about privacy regulations and consent for recording and processing of the data.
Iarossi (2006, p. 163 .) summarizes rules for the interviewer by thefollowing commands: 'be prepared, be likeable, be positive, be responsive, be
neutral, do not lie, do not threaten, do not rush, do not ignore'.
Similarily, the personality of the interviewee can have an impact on theinterview in dierent ways. A 'perfect' interviewee would be intellectually aswell as verbally competent, such that she is able to understand and reply toquestions.4 She would be genuinely interested in the interview topic and at
the same time neutral in replying. In reality, however, dierent factors are
known, which can possibly distort the answers: reactions to pre-assumptions
about the aim or responsible of the survey, the interviewee's desire to givethe 'right' answer (i.e. by guessing what is expected), the desire to present
oneself at his best, the desire and to act and reply consistently, as well as
general intimidation of the interviewee or a closed personality.

Guidance within the interviewing situation
As described above the relation between the interviewer and the interviewee
can have a considerable impact on the interviewing situation. For a prospective interviewing situation the interviewer should create an open, neutral andrespectful atmosphere, in which answers and explanations of the interviewee
should be encouraged by the interviewer in a neutral way, while judgements
and an atmosphere of examination should be strictly avoided.
Furthermore, 'it is important to communicate to the respondent how to
perform his or her role as information provider and explain the setting inwhich he or she will be asked to provide information' (Iarossi, 2006, p. 179).

In fact, an interviewee can enter into the interviewing situation with dierent
opinions or view points depending on which of her roles in society she is
4

The task of replying to an interview question requires the completion of ve actions:to understand, to recall, to decide, to formulate, and to reply (cf. e.g. Porst (2008, p. 17)
and Strack and Martin (1987)).

3.2. Methods of enquiry

71

assuming during the interview (e.g. a single person can at the same time
hold the roles of an ordinary citizen, a manager of a company, and a motherof a small child).
From an instrumental perspective it is advised to start an interview witha set of opening questions which are easy to reply to, create a relaxed atmosphere and spark the interviewees interest. During the course of theinterview, the interviewee is steered by neutral probing, that is asking forclarication or additional info in a nondirective way, or by prompting, thatis suggesting dierent answer alternatives (e.g. to make answering easier).
An interview is concluded with a number of closing questions which allow
for a smooth and positive closure of the interview, while possibly giving theinterviewee the opportunity to add any additional relevant information. Finally, the interviewee should be thanked and can be asked permission to be
re-contacted for clarications, if needed.

Planning and conducting interviews
For carrying out meaningful interviews a scientically valid methodology
should be followed, which requires a series of steps and decisions (cf. Helfferich (2005, p. 147 .)):1. denition of the overall research objective, concrete research questions
and the core topics to address within the interview
2. specication of the target group and sample
3. decision on the type of interview and targeted status of results (e.g.
exploratory insights, represenativeness for population XY, etc.)
4. translation of research questions into interview questions, and designof the interview guideline 5 ; Helerich (2005) provides detailed instructions on the subtasks involved:(a) brainstorming on any relevant questions(b) reduction of questions to a concise set (respecting question design
rules, section 3.2.1.1)(c) grouping of questions into 1-4 thematic blocks, chronological ordering(d) design of introductory questions for each theme(e) placement of o-topic questions at the end
5. conduction of a trial run of the interview ('instrumental pre-test')
6. contacting interviewees
5

Translation from German original 'Interviewleitfaden'

72

Chapter 3. User-centered design and evaluation
7. conducting and recording interviews
8. transcribing audio or video recordings for analysis

The interview guideline is the operationalization of the research objectives and serves as a standardization instrument for data collection. The
guideline documents the decisions taken regarding the interview structure
and topics, and thus ensures the consistency and completeness of the investigation over the entire research period. It thus guarantees the replicabilityof the procedure and the comparability of the interviewing results.

3.2.1.2 Questionnaires
Along with interviews, questionnaires are one of the most widespread survey method. Questionnaires show a greater degree of standardization thaninterviews. They usually have a highly structured format and are mainlycomposed of closed questions with predened answer categories.6 Accordingly, the preparation of a questionnaire requires in-depth knowledge on the
topic under enquiry in order to safeguard that the proposed answer choices
are plausible and exhaustive. Due to the high degree of standardization,questionnaires are particularly suited for the close and in-depth investigation on a well-dened subject.
Questionnaire surveys have the advantage to allow for a broad investigation with relatively low costs. Once the questionnaire design is completed,they neither require the direct intervention of a researcher nor the physical presence of the test person, thus they can easily be distributed to large
numbers of geographically spread participants. In addition, standardised answer categories allow for a fast processing of the collected data and related
quantitative analyses.
As questionnaires are usually compiled without direct guidance, it isnecessary that questionnaires are self-explanatory. They need to be carefully
structured and worded to avoid confusion and misunderstandings.
At the same time, questionnaires need to be designed and introduced ina way that encourages the participants to compile it up to the end (avoiding
drop-outs), even without having an accompagnying person motivating them.
Bortz and Döring (2006, p. 257) indicated several factors that can positively inuence participation rates: interest and relevance of the subject ofinvestigation for the participants, the introductory explanation for the questionnaire, the executing institution (public and research institutions tend toreceive higher participation rates than private institutions and companies).
Furthermore, Iarossi (2006, p. 147) indicates that the social environment(the perceived importance and legitimacy of the survey), the survey design,
6

If largely composed of open questions it would be considered a written interview.

3.2. Methods of enquiry

73

and the respondent's state of mind (e.g. desire for self-expression and altruism, as well as fear, and resentment towards the furtherer) can inuence
survey participation. As well, announcing a deadline for the compilation has
shown to have a positive impact, and little incentives (i.e. small presents)
have proven to increase participation (Iarossi, 2006).

Question types
Questionnaires can be composed of open, closed and semi-open questions. Open questions come with free text elds and pose no restriction
on the user's answer. Closed questions, to the contrary, present a set ofpre-dened answer options of which the respondent can choose from. Semiopen questions are closed questions with an additional text eld which gives
the respondent the possibility to provide replies which are not part of the
pre-dened set.
Open questions have the merit to allow the respondent to fully express
his point of view. At the same time they bear the risk to receive unclear
answers, short answers or no answers at all, as open questions are the mostdemanding to the respondent. For example, it has been observed that responses to open questions can be aected by the respondent's fear to produce
orthographic errors.
Closed questions, on the contrary, have the advantage to allow forstandardization which greatly facilitates the analysis of results. At the sametime they pose the challenge that, in order to work well, they need to provide exhaustive answer options. Incomplete or selective answer options canconfuse the respondent and lower the overall credibility of the questionnaire.
Furthermore, with closed question respondents have a tendency to stick to
the predened answer choices, rather than using the additional text eld.
A particular type of closed questions are rating questions, for which the respondent has to provide an evaluation. They can be presented as questions
or statements.7 .
Overall, it is advised to complement closed questions with sporadic open
questions, to allow for unexpected answers and to keep the questionnaire
more lively.

Answer types
Answers to open questions are collected in the form of free text.
Closed questions can oer single-select or multiple-select answer options. Multiple-select answer options are on a nominal scale, while answer
options to single-select questions can be nominal, ordinal or on interval
7
e.g. 'How important do you consider xyz?' with options 'very important', 'important',
etc. vs. 'It is very important to xyz' with options 'I fully agree', 'I agree', etc.

74

Chapter 3. User-centered design and evaluation

scale. Rating questions are a special case of single select questions, which
typically make use of a Lickert scale for requesting an evaluation.
Answer options on ordinal or interval scale can be spelled out in words8
or be presented as a scale9 .
The ideal number of answer options for a Lickert scale is an open pointof discussion. On the one hand, an equal or uneaqual number of options
could be preferably, i.e. an unequal number bears the risk to receive many
middle-value replies, while an equal number forces the respondent to decide,which might not always be adequate. On the other hand, it is not clear whatshould be assumed the most adequate width of a scale. A wider scale allows
for more ne-grained distinctions while it might overburden the respondentand be cumbersome for analysis.
Regarding answer options on ordinal scales, studies have shown that thecomposition of the set of options oered can have a considerable impact on
the responses.10
In case of multiple-select questions, especially for questions with long listsof answer options, primacy and recency eects must be considered. That is,the rst elements of a long list tend to get unproportionally higher numbersof selects.
Finally, Iarossi (2006) suggests that within a single questionnaire thevariability of dierent question types should be limited in order to minimize
confusion. In the same way, the type and lengths of scales should be kept
consistent, as well as their orientation (e.g. positive values always to the leftof the scale).

Question design
Concerning the design and wording of questionnaire questions, generally thesame rules apply as for interview questions (see Section 3.2.1.1). However,given that questionnaires are compiled without the assistance of a researcher,
it is even more important to safeguard that questions are clear and easy tounderstand. And as stressed by Fowler (1995, p. vii) 'improving question
design is one of the easiest, most cost-eective steps that can be taken to
improve the quality of survey data'.

Regarding the design of closed questions, which are typically prevalent
in questionnaires, it is important to provide comprehensive and plausible
answer options which should be mutually or collectively exclusive. Also, it
8e.g. 'very important', 'important', 'somewhat important' 'not important'e.g. '1 = very important', '2', '3', '4', '5 = not important' with labeled end points
10
Porst (2008, p. 61) reports on an experiment by Schwarz et al. (1985), for investigating
on average daily times for watching TV. They could show a considerable dierence in thenumber of people who indicated to regularly watch more than 2.5 hours of TV, dependingon whether the scale had the 2.5 hours item placed on the lower or upper end of the scale(i.e. 'up to 2 hours', '2 to 2.5 hours', 'more than 2.5 hours' vs. 'up to 2.5 hours', '2.5 to
3 hours', '3 to 3.5 hours', 'more than 3.5 hours' )9

3.2. Methods of enquiry75

is advised to avoid extreme terms like always, never, or nobody, as they arelikely to be perceived as implausible, as well as vague terms like sometimes,
almost or often as they cannot reliably be compared across respondents.
Furthermore, it has been observed that respondents tend to select predened answer options even for subjects that they are not knowledgeable
about.11 Considering eects like this, it is advisable to carfully estimate therespondent's level of knowledge, and to consequently avoid questions about
topics on which the interviewee might not be competent or informed.

Questionnaire structure
The design of a questionnaire, in terms of the overall information ow andguidance of the respondent, as well as regarding the content and wordingof questions, is decisive for the acceptance and reliablity of this survey instrument. In fact, too long, too hard to understand and too unprofessionalquestionnaires have shown to irritate the user and to often receive lower
response rates (Nielsen, 1993, p. 212).
First of all, a questionnaire should be presented with a clear and easyto-read layout. It should open with a motivating explanatory introduction,
which welcomes the participant, introduces the objective of the study as wellas the institution and person in charge. Furthermore, it should give instructions for the compilation of the survey, including its expected duration.
Questions should be grouped into topics, and the overall order of topics should follow a logical ow. Jumps between topics should be avoided.
Transitory sentences and explanations as well as subheadings can help to orientate the respondent. Filter questions can be used to guide the respondent
around irrelevant questions.
Introductory questions should be easy to answer, interesting, and related to the topic of the survey and the respondent himself. For example,
socio-demographical questions are usually placed at the beginning of a questionnaire (Bortz and Döring, 2006, p. 256). To the contrary, questions which
are delicate or dicult to answer should generally be place towards the endof the questionnaire, as they bear the risk of drop-outs.
It is suggested to conclude the questionnaire with easy to answer andpleasant nal questions, possibly asking for additional remarks on the topic
of study. Furthermore, the last page of the quesionnaire should include athanks to the respondent for his participation and possibly give information
on how results of the study will be distributed.
11
Porst (2008, p. 122) cites experiments demonstrating that people recurrently provide
ratings on ctional politicians, e.g. when presented in lists with real politicians.

76

Chapter 3. User-centered design and evaluation

Planning and conducting questionnaire surveys
As with interviews, questionnaire surveys require a careful planning andimplementation in order to guarantee their succesful application and thescientic validity of results. The following steps are involved in questionnaire
survey preparation (cf. e.g. Iarossi (2006, p. 10 .)):1. review of literature and communication with experts to dene concrete
research objectives
2. translation of research questions into survey questions and constructionof a sound questionnaire with regard to question design and selection:(a) for each question its objective as well as the required type of
information needs to get dened(b) the relevance and adequacy of questions should be checked, where'a question is relevant when the information generated is appropriate for the purpose of the study' and 'a question is accurate if

it collects the information sought in a reliable and valid manner'(Iarossi, 2006, p. 27)(c) to avoid lengthy questionnaires only questions which are of substantial relevance for the research study should be included

3. design of questionnaire layout and set-up of questionnaire form, or
print-out, including lters and consistency checks (for computer forms)
4. piloting of the questionnaire with prospective respondents to rule out
unclear parts and misleading formulations
5. selection of the sample of participants
6. contacting and inviting people for participation
7. running the survey
8. coding of survey data into categories for analysis, e.g. open or continuous answer options are grouped into discrete answer categories

3.2.2 Data analysis methods
Regarding methods for data analysis a distinction is made between qualitative analyses and quantitative analyses, depending on the type ofdata that is analyzed and the types of insights that ought to be derived.
As detailed in the previous subsection interviews and questionnaire surveys
allow to collect various types of data, including free text and predened answer options (nominal, ordinal, interval). Depending on the open-endednessof the research question analysis methods can serve an exploratory purpose

3.2. Methods of enquiry

77or allow for the testing of hypotheses. In the following subsections analysis
methods relevant for the analysis of interview and questionnaire data arepresented. The descriptions of the data analysis methods are based on Bortz
and Döring (2006), Mayring (2003), and Mangiaco (2016).

Exploratory qualitative data analysis is the open-ended and impartial
analysis of data in order to approach the theoretical description of
a subject by means of determining, listing and structuring relevant
aspects of the data related to it.
Qualitative content analysis is a technique for analyzing texts by dividing each text into units, grouping the information contained withineach unit with regard to a pre-dened set of analysis categories, andsubsequently summarizing repetitive information and describing it ona higher level of abstraction.
Descriptive statistics are quantitative descriptions of a data set by meansof summative calculations, typically including measures of central tendency (e.g. mean, median, mode, etc.) and measures of dispersion(e.g. standard deviation, min/max values, skewness, etc.).
Signicance tests are any type of calculations regarding the statistical relevance of observed dierences or similarities between groups. They
include tests like 't-test', 'Chi square', 'Wilcox', 'Kruscal-Wallis', etc.
3.2.2.1 Exploratory qualitative data analysis
Exploratory qualitative data analysis (Bortz and Döring, 2006, p. 381
.) is an open-ended approach to the description of data. It aims at buildinga rst understanding and theoretical basis of the underlying subject by analyzing related data in its entirety without relying on predened theoreticalrules or assumptions.
Bortz and Döring (2006) name dierent activities which can be part ofthis exploratory analysis process. A rst step in approaching a subject consists in determining its most relevant aspects and elements and listing themas comprehensive inventories. The follow-up step typically aims at orderingand grouping the elements of these inventories into types or structures whichshare characteristics or 'behave' similar on a higher level of abstraction. Another important step for theory building is the detection and descriptionof relations and interdependencies between phenomena and their respective
causes. In addition to inventories, also an overview of the involved processes
is often relevant, and can be derived from the close analysis of case studies.
Taken together, these approaches to the exploratory analysis of data leadto the creation of a comprehensive and partly formalized description of thesubject of study.

78

Chapter 3. User-centered design and evaluation

More specically, exploratory qualitative data analysis is employed
to analyze text material in an exploratory fashion to detect interesting aspects in the data and group information on a coherent topic together, without referring to a specic predened theory, but in order to generate theory
from the data. This explorative and open-ended analysis method is particularly suited to analyze highly unstructured data with no pre-dened analysis
scheme, as can typically result from open interviews or eld studies.

3.2.2.2 Qualitative content analysis
The qualitatitve content analysis (Mayring, 2003) is a systematic method
for the qualitative analysis of text, which relies on a set of rules to successively
analyze a text based on a theoretically-motivated category system.
The qualitatitve content analysis responds to the demand for systematic and replicable methods for the analysis of qualitative data, which
has increased since methodological developments in the social sciences andhumanities have led to a wider acceptance and inclusion of qualitative approaches.
While complex quantitative methods deliver measurable and replicable
results, they pose strict requirements onto the data under analysis (e.g. exhaustive categorization, sample size, standardization of data sets etc.) andthus limit the openness of an investigation (i.e. questionnaires with predened answer categories). To the contrary, open approaches to data collection(e.g. open interviews, focus group discussions, etc.) result in less structured
data, which however can contain complex information and information which
has not been anticipated. These methods of data collection are thus particularly relevant for theory and hypothesis generation, pilot studies, construction of descriptive systems (classication systems), etc., but pose challenges
for data analysis, in particular the challenge to provide comparable and nonsubjective analyses of the resulting qualitative data.
With the qualitatitve content analysis a traceable and replicable
methodology for the systematic interpretation of qualitative data has been
introduced. It relies on an initially dened category system as core reference
for analysis and three basic analysis operations, which are applied side-byside: (I) summarization, (II) explication and (III) structuring. First of all,the text under analysis is subdivided into analysis units and subsequently
the analysis procedure is applied unit by unit in an exhaustive fashion.
Summarization aims at reducing the textual material to its essential content by means of abstraction. It consists in several steps: paraphrasing ofeach analysis unit, generalization of the content to the targeted level of abstraction and reduction to non-redundant statements, integration of several
generalized statements to a newly constructed statement and nal crosschecking if generated statements represent the original information content.
The summarization procedure can be iterated to successively reduce the ma-

3.2. Methods of enquiry

79

terial to higher levels of abstraction.
While the summarization step is typically oriented on the initially dened category system, it can also be implemented in an inductive manner inorder to generate categories from the text material. This procedure is called'open coding' and relies on theoretically-informed and thus more abstract
selection criteria. While coding along a initially dened category set results
in data ready for interpretation, the category set resulting from the inductive
summarization procedure can be directly interpreted regarding the researchquestion, reworked to dene main categories for analysis, or quantitatively
evaluated with regard to its textual instantiations.
Explication aims at further describing unclear parts of the text by thehelp of additional material (close or wide context). The explication process
starts from the lexico-gramatical structure of the text and denes the additional addmissive data for interpretation. Based on the additional material a
formulation will then be seeked that best explains the unclear text segment.
Finally, it will be double-checked whether the explication is reasonable.
Structuring consists in the ltering, investigation and evaluation of the
textual material regarding dened aspects such as formal text characteristics, general content aspects, type-related content aspects or the evaluation
regarding specied scales. The structuring is implemented by assigning eachtext segment to one of the dened categories following pre-dened coding
rules.
Overall, the qualitative content analysis includes the following steps(Mayring, 2003, p. 46 .):1. denition of the base material for the analysis
2. description of the context in which the material was produced
3. formal description of the material12
4. denition of the research objective(a) target of analysis13(b) reference to a particular theoretical framework
5. decision and denition of which analytical techniques and process model
are applied(a) analysis units (mininum and maximum size of units, order inwhich units are analyzed)(b) category system (is developed with reference to theoretical background and to the material for analysis; adjusted and re-evaluated/conrmed
during analysis)
12
13

E.g. 'transcription of interview recordings with reference to transcription rules xyz'
E.g. 'description of content or of emotional state of author'

80

Chapter 3. User-centered design and evaluation(c) rules for assignment of categories
6. analysis using summarization, explication and structuring
7. interpretation of results

3.2.2.3 Descriptive statistics
Descriptive statistics allow to create a summative overview of data samples (Bortz and Döring, 2006, p. 361). Descriptive statistics include absoluteand relative frequencies, frequency distributions, values of central tendency(mean, median, mode), dispersion, skewness, cross tabulation and correlation matrices. Descriptive statistics can be displayed in numerical form (e.g.,
data tables) or as graphical visualizations (e.g., histograms).
With regards to results from questionnaire surveys, descriptive statistics
for single answer options and Likert scale data are relevant. Single answer
options can be indicated as absolute and relative frequencies. Likert scaledata is commonly considered as ordinal data and not as interval data, since
it cannot be assumed that the dierent values on a Likert scale are equally
spaced (Mangiaco, 2016). Accordingly, the primary measure of central tendency is the median. Mean and standard deviation cannot be used without
specic justication and the distribution is best captured by visualization asa bar plot.
Statistics software packages oer functions for calculating (and partly
visualizing) descriptive statistics easily.
3.2.2.4 Signicance testing
A wide variety of statistical tests of signicance is available for dierent
analyses. Dierent statistical tests pose dierent conditions on the data
type, sample size, distribution, etc. Given that a comprehensive overview
of dierent tests of signicance would greatly exceed the space limits of thissection and unnecessarily repeat the information found in relevant textbooks(cf. e.g., MacClave et al. (2000); Witte and Witte (2017); Freedman et al.
(2007)), we will focus on introducing the Mann-Whitney-U-Test14 , which issuited for Likert scale data.
As explained above, Likert scale data should be considered ordinal scaledata. Therefore, non-parametric tests need to get applied for calculating
signicant dierences between samples. The two-sample Mann-Whitney-UTest is a rank-based test, which is adapted for ordinal data (cf. (Mangiaco,2016)). It compares values of two groups, by assessing 'if it is likely thatan observation in one group is greater than an observation in the other'(Mangiaco, 2016). The samples have to be independent and no normal
distribution of the values is required.
14

also known as Wilcoxon Rank Sum text

3.3. Challenges in usability research81

The hypotheses are formulated as follows (cf. (Mangiaco, 2016)):
 Null hypothesis: The two groups are sampled from populations withidentical distributions. Typically, that the sampled populations exhibit
stochastic equality.

 Alternative hypothesis (two-sided): The two groups are sampled from

populations with dierent distributions. Typically, that one sampled
population exhibits stochastic dominance.

The test indicates if the dierences among the values of the two groups
can be considered signicant. It does not give any indication about thedirection of the dierences (i.e., group A is greater than group B). The
direction of the dierences is found by comparing the medians.
Statistics software packages oer functions for applying the Mann-WhitneyU-Test to any dataset. It is equivalent to a two-sample Wilcoxon rank-sum
test as well as to the KruskalWallis test with only two groups (Mangiaco,2016).
3.3

Challenges in usability research

HCI provides general approaches for human-centered software evaluation.
However, the successful design and evaluation of a specic service or tool
is not only dependent on following a sound usability research methodology
but also very much relates to the application domain and objectives forthe software. This results from the overall principles of usability which take
into account the user and her individual characteristics, the goals and tasks
that are pursued, as well the related environment of use (cf. Section 3.1.1).
Consequently, the more conned and predictable these variables are for a
specic application domain, the easier it should be to determine and address
usability requirements for related services. To the contrary, heterogeneoususer populations and usage environments, as well as varied and openended goals and tasks can pose major challenges for usability designand evaluation, a situation which in fact concerns various domains, including corpus linguistics. The major challenge for corpus linguistics is thevariety and open-endedness of tasks related to dierent research objectives(cf. Chapter 2, Section 2.3), as well as varying education and skill sets ofthe corpus users.
In order to get an idea of the types of challenges which can occur
and the attempts that have been made for addressing them, three elds
which provide tools for open-ended (non-deterministic) enquiry procedures
are looked at. In particular, the following subsections are related to discussions about how user-centered research objectives can or should be tackled

82

Chapter 3. User-centered design and evaluation

regarding: specialized dictionaries (Section 3.3.1), search and discovery systems (Section 3.3.2) and web search interfaces (Section 3.3.3).
Similar challenges apply to other elds and the reported work is to be understood as exemplarily. However, the value of these mainly older works istheir particular active scientic discourse on usability methodology,
which elaborates on key issues in usability research for open-ended domains.
The challenges and proposed solutions are summarized in Section 3.3.4
and are partly retaken in Section 3.4 when concluding on this chapter andits implications for the study design.

3.3.1 User-centered research in specialized lexicography
As proposed by Heid (2011), questions of usability are also applicable todictionary design and evaluation. Indeed, several experts understand anddescribe (electronic) dictionaries as tools (cf. e.g. Tarp (2008), FuertesOlivera and Bergenholtz (2011)).
In particular Bergenholtz et al. (1995) started addressing usability questions in lexicography early on. Initially, the design of specialized dictionaries
has been approached within the context of meta-lexicography, which builds
on three pillars (Bergenholtz et al., 1995, p. 31):
1. user research: formulation of theories, methods and models for examining and enhancing dictionary use
2. dictionary criticism: dictionary reviews and models for the assessmentof dictionaries
3. systematic dictionary research: formulation of novel or enhanced theories with the objective of their adoption within dictionary design processes
The authors discuss the information needed to determine requirements
for dictionaries. More specically, they stress that 'every specialized dictionary is compiled with a certain user type in mind' (p. 20) and claim that'user research should form part of the preliminary work on every new dictio-(p. 232). They reect on the viability of user surveys by discussing
dierent survey methods (p. 77 .).
In later works, Bergenholtz and Tarp (2003) dene the genuine purposeof every dictionary as follows: to serve a specic (information-seeking) purpose and to respond to specic requirements connected to it. Accordingly,
Bergenholtz and Tarp (2005) suggest that all lexicographic work should bebased on the dictionary functions which represent the assistance provided tothe dictionary user with regard to three variables: (1) the type of user, whichis addressed, (2) the type of user situation and (3) the types of problems related to this user situation, which aligns well with the general evaluation
criteria for usability, as discussed in Section 3.1.1).
nary'3.3. Challenges in usability research

83

Improving and enhancing the design of dictionaries thus should rely on: a clear denition of the purpose and target group of a dictionary, requirements collected from extra-lexicographic user situations (com-munication and knowledge related), and accordingly, systematic descriptions of the base-categories of lexi-

cography (user, situation and requirements), which provide an abstraction over dierent usage situations.

In order to anticipate new classes for dictionaries the authors consider it
crucial to analyze extra-lexicographic user situations instead of reconstructing needs from existing dictionaries. More specically, it is expected that
evaluating existing solutions is likely to miss out on new information as
the users' expectations and that previous experiences can bias their lookup strategies. At the same time, Bergenholtz and Tarp (2005, p.17) notethat requirements analyses based on extra-lexicographic situations are timeand cost intensive, and pose yet unsolved methodological and interpretative
challenges.
Summing up, the user, his usage situation and related tasks and requirements need to be taken into account to determine relevant design criteria
for dictionaries. Furthermore, the design process needs to focus on extralexicographic usage situations instead of on the evaluation of existing dictionaries only, in order to introduce novel features. Overall, the open-ended
analysis of users' needs is considered the best approach to dictionary design,
while at the same time, it is expected to be time-consuming and methodologically challenging.

3.3.2 Design and evaluation of knowledge discovery systems
Search and discovery systems aim at supporting users in information
seeking tasks. By their very nature as retrieval tools they are typicallyhighly interactive and strongly steered by the user.
Initially, the evaluation of discovery systems mainly focussed on theperformance and accuracy of their underlying algorithms. Later on, also theuser-centric evaluation of these systems has become of wider concern. This
has led to intense discussion about applicable and meaningful approaches
to evaluation for these specialized contexts (cf. e.g., a dedicated LREC
workshop15 on the topic).
Underwood (2004, p. 18) denes the goal of Knowledge Discovery(KD) as 'the uncovering of [] information from data [] which was previously
15

Pre-conference workshop 'User Oriented Evaluation of Knowledge Discovery Systems'
at LREC 2004, Lisbon, Portugal; see http://www.lrec-conf.org/lrec2004/doc/ws/useroriented.pdf

84

Chapter 3. User-centered design and evaluation

not apparent or readily accessible because of the amount and complexity of
data'. Knowledge discovery systems (KDs) typically incorporate data mining

and intelligent information extraction techniques. KDs are 'intelligent tools
for qualitative analysis that support, rather than substitute, the user in the
exploration of the data' (Stuckardt, 2004, p. 24). KD systems produce sta-tistically indicative and therefore preliminary results, which require further
human processing and interpretation to complete higher level tasks. These
properties of KDs and KD tasks result in several challenges for their formal
evaluation, as traditional evaluation metrics are often not applicable.
One principal issue with evaluation is the lack of any direct correlation
between the performance of the NLP technology and the perceived usabilityof a system. Thus testing of system attributes does not reliably allow forpredicting the utility of a KD system (Underwood, 2004).16 This issue is related to 'the qualitative gap' (Stuckardt, 2004, p. 26), which occurs aroundsoftware solutions for intellectual data exploration, where practical limitations of technology for automatically identifying relevant content inevitably
leave part of the task of 'gaining insight' to the discretion of the user.
Another issue is related to the diculty to formally dene a prototypicaltask which matches the knowledge discovery needs of all (Stuckardt, 2004).
Task descriptions of the user may be rather complex since the overall goal
may comprise a number of smaller sub-goals that the KD system should help
with, and therefore goals of users often 'can only be dened at the highest
level of abstraction in terms of the desire to gain new insight from data'(Underwood, 2004, p. 19). According to this high complexity of KD tasks
and to the large amounts of data that KDs are dealing with it is not possible
to predict the expected correct results. It is thus unfeasible to produce gold
standard data and to apply the traditional evaluation metrics 'precision' and'recall'.
Furthermore, there is the diculty of dening a typical user of a KD
system, as overall goals and the concrete results that users are looking forcan be very varied (Underwood, 2004). In addition, users may also dier
regarding their level of experience and their expectations.
Underwood (2004) suggests to approach the evaluation of KDs by asking users for judgements along a predened set of 'soft metrics' (novelty,
credibility, relevance, understandability, and subsequent use of results), andby clearly distinguishing user types regarding their role, their level of expertise in the subject domain, their experience with similar systems, their
expectations and assumptions towards the system.
Stuckardt (2004) suggests to approach evaluation by constructing 'best
practice guidelines'. Oriented on suggestions within the domain of spoken
dialogue systems these guidelines should provide 'a mapping from functionalparameters to parameters of design and development' (van Kuppevelt et al.,16

E.g. even if a trend is statistically relevant, it may not be relevant for the user.

3.3. Challenges in usability research

85

2000, p. 207) and should cover:1. the stock-taking of the state-of-the-art in order to enable the stakeholders to quickly get informed about available options for design, implementation, and evaluation,
2. quality control through criteria that support the selection of the best
suited options for particular application requirements, and3. economic control through the creation of a repository of resources tofoster the reuse of components, design know-how, and experiences.

3.3.3 User-oriented evaluation of interactive web search interfaces
Hoeber and Yang (2007) discuss challenges of user-oriented evaluations regarding interactive web search interfaces. Since they 'are designed tosupport exploratory searching, providing a range of actions that are much
more complex than the common list-based representation of search results',

they state, 'many of the simpler techniques that have been used to evaluate

static Web search interfaces are not as eective for the evaluation of intelligent and interactive Web search interfaces' (p. 242).

They approach the challenge by discussing how three relevant aspects forevaluation have been tackled in related works: the experimental design,
the task assignment, and the measurements. They show that the experimental design, that is the setup of the evaluation condition, has to becarefully chosen so as to avoid ambiguous or biased results. For example,
when comparing several systems, the order in which systems are tested canhave an impact if it cannot be prevented that participants transfer information they gained within the experiments from one system to the other. They
discuss the eects of pre-dened search tasks vs. search tasks freely
chosen by the user, where xed tasks allow for a comparison of dierent
users, while they force the user into an articial and not authentic task, forwhich the entire interaction might be result in a limited informative value.
The third issue for evaluation concerns the choice of valid, meaningful andcomparable measurements. Relevant parameters for evaluating search systems include whether a task can be performed faster, more accurately, withless errors, with more satisfaction and with higher condence. The parameters speed, accuracy and error rate require relevance judgements, whilesatisfaction and condence can be measured with questionnaires collecting opinion statements.
Hoeber and Yang (2007) report on dierent types of relevance judgements on search results which are found in the literature: logs for documents
viewed as indicator of their relevance, binary relevance judgements requested
from the user, relevance judgements on a nite or continuous relevance scale,

86

Chapter 3. User-centered design and evaluationand top ranked search results by requesting the user to create a ranked or-dered list.

For measuring the time to completion clear completion criteria are
required. Proposed solutions include: users may search until they decide to
be satised, users have to consider a limited number of documents and choose
the relevant ones, users have to indicate a predened number of relevant
documents (by considering as many documents as they want).
Another issue mentioned, is that for the collection of opinion statements (subjective measures) it has to be decided when to request thejudgements. It has been suggested to request evaluation of general featuresat the very end of an analysis session, while evaluation of specic tasks canbe requested after the completion of every single task.
The option to tightly integrate the evaluation procedure with theanalysis workow ('automatic data collection') bears the risk to create
interference with the completion of primary tasks, e.g. by deviating theuser's attention from the search task to the evaluation task (cognitive loadand distraction; change of focus of attention). The solution suggested by
Hoeber and Yang (2007) is to use verbal protocols, which are created by ausability evaluator recording the users comments. It is expected to be least
intrusive and at the same time to allow the usability researcher to carefully
watch the user performing the task, and meanwhile gaining further insights
on how the user carries out the task.
They conclude that '[c]onducting user-oriented evaluations of intelligent
and interactive Web Search interfaces is dicult due to the high degree ofinteractivity promoted by such systems.', and observe that 'while numerous

studies have been conducted [...] there is little consensus on the proper methods to use' (Hoeber and Yang, 2007, p. 243).

3.3.4 Summary of challenges and proposed solutions
The discussion in the previous sections shows that several common challenges
are encountered in the user-centered design and evaluation of systems forthe discovery of insight and knowledge from textual data.
In the rst place, user proles and related usage situations show ahigh variability, which leads to a diversication of the overall user needs
to which existing tools might not respond in the best possible ways.
Second, the open-ended objectives in the discovery of knowledge posethe diculty of dening clear goals or targets for the software system. Also,
there is a high variability in tasks across dierent users and tasks areoften complex compositions of several subtasks. Therefore, it is dicult todene general tasks that are applicable to the needs of many users andto depict their expected results. Both aspects are crucial conditions
for evaluation. Clear task descriptions and results are for example needed
to create gold standard data which allow to calculate precision and recall

3.3. Challenges in usability research

87

values for experimental results.
Finally, systems for open-ended discovery tasks are a support technology rather than a fully automated tool. The tools provide assistance tosearch and analyze data, but the conclusions drawn by the user go beyond
the results delivered by the tool. Stuckardt (2004, p. 26) describes this as thequalitative gap around software for intellectual data exploration, where technical limits of fully automating the content extraction leave part of the taskof 'gaining insight' to the discretion of the user. This results in a situation
where the performance of the system (in terms of NLP technology) does
not equate with its overall perceived usability. These conditions greatly
complicate user-centered design and evaluation. The related works presented
in the previous sections come forward with suggestions for adapting both thefocus and the techniques of user-centered evaluation.
On the one hand, strong concern needs to be given to understand thecharacteristics of the users and their objectives. Understanding theobjectives of the users is linked to their '(extra-lexicographic) user situation' and their 'information-seeking purpose' as Bergenholtz and Tarp
(2003) and Bergenholtz and Tarp (2005) put it. This means that the goals
and tasks that a user aims to achieve should be observed detached from
specic tools for achieving them. Evaluating requirements in relation toexisting tools and tool functionalities bears the risk to introduce bias towards
existing solutions and hinder the identication of requirements that derivefrom the core objectives and working procedures of users.
On the other hand, user-centered studies should aim to identify genericrequirements. Given the variability and complexity of tasks and data thisis far from being trivial. Bergenholtz and Tarp (2005) suggest to start fromthe base categories of a domain to abstract away from idiosyncratic usage situations. Stuckardt (2004) propose to evaluate systems against 'best
practice guidelines' that is, criteria for choosing the best suited options,which are built on mappings from functional parameters to parameters ofdesign and development. Underwood (2004) suggests to gather user judgements on 'soft metrics' like novelty, credibility and relevance of results.
Evaluations that focus on system attributes only are considered problematic. As explained above, system performances do not necessarily correlatewith the use of the tool, especially in the case of supportive tools.
Given the open-endedness of tasks and objectives, evaluation in experimental settings poses two major obstacles. On the one hand, proposing
unique predened tasks to all users bears the risk of creating use cases thatare not authentic, while letting users choose their tasks individually leadto results which are not comparable across participants. On the other
hand, hard measurements like speed, accuracy and error rate can only beapplied in relation to relevance criteria, which again depend on the target
and expected result for a task. Also, when comparing systems, the order inwhich alternative systems are presented can lead to inferences amon the re-88

Chapter 3. User-centered design and evaluation

sults for dierent systems. And when collecting judgements on tool features
the timing of asking for the judgements can inuence the performanceof the task completion17 . Hoeber and Yang (2007) conclude that verbal
protocols , which are created by a usability evaluator who is recording usercomments are the most reliable way of collecting observations and opinionsfrom the user, at the cost of being a time consuming method.
From the overview of common challenges that emerge in dierent related
domains, we can conclude that the user-centered evaluation of systems foropen-ended knowledge discovery is still an unresolved issue and potential
solutions to it are both dicult and time/cost intensive.
3.4

Conclusions and approach of this thesis

This chapter gave an overview of human-centered approaches to enquiry,
and software design and development. First, the objectives and general
concepts of human-centered research and development have been introducedin Section 3.1. Second, established methods of user-centered enquiry andevaluation have been presented in Section 3.2. And nally, challenges foruser-centered research on systems for open-ended discovery processes
and considerations for how to overcome them have been elaborated based on
discussions from related domains in Section 3.3.
Relating the presented background information to the research objectives of this thesis, we can observe that the overall aims of user-centeredresearch (Section 3.1) are in line with the demands for user-centered research in corpus linguistics (cf. Section 1.1). This holds both for generalneeds towards the corpus linguistics approach and its methods and procedures, as well as for specic requirements related to corpus linguistics
software tools.
Also, the overview in Section 3.2 conrms that a wide range of enquiry
methods are available for user-centered research. The focus of Section 3.2
on methods for interview and questionnaire investigations and their analysis
results from the decision to carry out the studies of this thesis by means ofinterview and questionnaire survey inquiries, as motivated below.
Indeed, the study design has been dened in response to general challenges for the user-centered evaluation of corpus research procedures
and tools, which address a particular type of process for insight andknowledge discovery from textual data.
As described in Chapter 1, throughout the seventy years of corpus linguistics' history software tools for corpus linguistics have remained mostly
17

I.e., if the judgement is collected after the task completion the user might not recall
everything he noticed, and if the judgement is collected while the user performs a task,
putting the additional cognitive load of answering questions might inuence the users
performance in completing the task.

3.4. Conclusions and approach of this thesis

89

unchanged in terms of their core functionalities. And aside from various
technical improvements, advancements on the functional level are very modest up to today (cf. Section 2.5.3). This apparent hold-up of progress in corpus tool development goes together with a sparseness of user-centeredand usability research in the corpus linguistics domain (cf. Section 2.6)
and makes the observations from conceptually similar areas of research become particularly relevant.
The open-ended nature and high variability in research objectives
and user proles in corpus linguistics and the corresponding challenges foruser-centered studies could be part of the explanation for this situation.
Indeed, the challenges highlighted in the discussion in Section 3.3.4 translate
directly to the corpus usage scenario.
User-centered approaches to corpus linguistics methods and tools haveto account for the following reality of the corpus usage scenario. Corpus
studies deal with a high variety of research objectives with open-ended
targets, which are approached by analyzing large amounts of complex
data. Corpus tools serve as support technology to facilitate data analysis
and interpretation. While they are fundamental for the feasibility of any
corpus study they do not deliver nal responses to any linguistic researchquestion (qualitative gap ), but serve the corpus linguists in retrieving therelevant data for interpretation (cf. Section 2.2). Last but not least, thevariability of corpus user proles, in relation to their experiences, skills
and expectations, poses another hurdle to investigating on user needs, as
the needs of dierent user types can be considerably dierent, as will be thevariety of tasks.
The research objective of this thesis and correspondingly the design ofa user-centered study on corpus research practice and tools has been informed by the discussions of these related domains. In the rst place, thestudy aims at gathering information on the corpus linguistics usage situation instead of evaluating existing tools or specic system attributes.
This choice is motivated by the claim that evaluation in relation to existing solutions might introduce a bias or limitation on the needs expressed byusers. Since little systematic and formalized knowledge on corpus linguistics
working procedures and objectives is available the study starts with an inquiry about the base categories (basic building blocks) and 'best practiceguidelines' in the form of theoretical principles relevant for corpus linguistics research. This way the study also aims at deriving generic requirements by abstracting away from specic research objectives of individual
corpus users. To account for the open-ended objective of the study, semistructured qualitative expert interviews are chosen as the method ofenquiry for this rst part of the study. Qualitative interviews allow to collecta detailed picture of working realities in corpus linguistics and does not limit
the respondents to a set of predened answers.
Second, the study aims at evaluating corpus usage practice and needs in

90

Chapter 3. User-centered design and evaluation

relation to dierent user proles and for a larger number of respondents.
Therefore, the second part of the study is designed as structured questionnaire survey that builds on the results of the qualitative expert interviews.
This way informally established 'best practices' in corpus work and mappings between objectives and methods to approach the objectives
that are derived from the interviews can to a certain degree be quantiedwith a larger user base, and the variability between dierent user proles canbe measured. For the analysis of the survey results users are distinguishedby dierent characteristics related to their professional background, level ofexperience with the corpus linguistics approach and skills in scripting andprogramming. Both in the interviews and questionnaire survey, requirements
towards corpus tools are addressed and interpreted in relation to their role assupport technology rather than as fully automated analysis services, thus
as technical component within an intellectual analysis and reasoning process.
Soft metrics like relevance, frequency of use and applicability, are used tocollect judgements of corpus users based on their individual experiences with
corpus work.
Finally, the study design deliberately avoids any experimental evaluation of specic use cases, tools or tool features. This choice is motivated in
two ways: On the one hand, related works have suggested to detach usercentered enquiries from existing tools as they are likely to introduce abias or limit the exploration of new ideas and analysis paths. Even more so,
as on the other hand an understanding of the base categories and basic
building blocks of corpus linguistics, related needs and the impact of different user characteristics on them are still missing, and would be needed to
be able to design and perform meaningful experiments that go beyond theevaluation of fragmented and likely arbitrary solutions. Summing it up, experimental user studies are relevant for evaluating new functionalities, while
this theses addresses the necessary prerequisite for designing meaningful new
functionalities, which is a detailed understanding of the users' actual needs.

-

Chapter4-

Study on the usage of corpora for linguistic
analyses - interviews

91

92

Chapter 4. Interviews on Corpus Working Procedures4.1. Aim of the interviews

93

Aim of this chapter

This chapter aims at presenting the rst two steps of the user-centeredinvestigation of corpus linguistics research practice, which are implemented by two series of interviews with corpus users. It describes thedesign and setup of the interviews and reports on their analysis procedureand results. Also, it summarizes how the results informed the survey design.
The chapter starts with the aim of the interviews (Section 4.1). It
continues by detailing their setup (Section 4.2), including the target groups,
the interviewing method and the interview topics, as well as the analysis procedure and the scope of the results. Then it presents a detailed elaborationof the interview results (Section 4.3) in relation to user characteristics,
linguistic research subjects, corpus research procedures, management of corpus data, corpus tools and corpus methodology. The chapter concludes witha short summary including implications for the survey design (Section 4.4).
4.1

Aim of the interviews

Two interview-based investigations are the rst part of the user-centeredstudy. The interviews aim at gathering a rst general overview of corpus
linguistics research practice in order to build an explicit and comprehensive description of its current state and limits from a use-based view.
This requires to dene and distinguish the relevant key dimensions ofcorpus research practice and to describe and exemplify them by gathering
the variety of their instantiations from responses of the users and theirexperiences with corpus work.
By identifying the key dimensions of corpus work and their instantiations,the interviews in particular also serve to build the contextual ground forthe questionnaire-based survey.
In order to take an open view on the subject, the interviews are explicitly investigating on individual perspectives of individual users towards
corpus work. The rst set of interviews (I1) takes a case-based view on
concrete corpus studies and focuses on the implementation of workows,
and the use and limits of tools. It is followed by a second set of interviews (I2)which takes a more abstract and high-level view on corpus analysis practiceby shifting the objective from enquiring about concrete corpus studies andtools usage towards investigating on the methodologies in corpus linguistic
analyses and the canonical building blocks within a corpus analysis situation in terms of theoretical principles, data and tasks. Taken together,by means of the interview investigations a comprehensive descriptionof key dimensions in corpus research practice are derived from reports on
partly idiosyncratic analysis procedures of specic studies (I1) as well as from

94

Chapter 4. Interviews on Corpus Working Procedures

abstracted information on user experiences with the analysis usage scenario
in general (I2).
4.2

Setup of the interviews

4.2.1 Target group
The interviews targeted researchers doing linguistic studies based on
corpora. For the rst series of interviews (I1) corpus linguists with varying
degrees of experience and varying computational skills were selected, while
the second series of interviews (I2) explicitly targeted highly expert corpus
users.
The corpus linguists participating in the rst series of interviews (I1) werechosen based on an analysis of submissions to the referenced bi-anually Cor1
pus Linguistics Conference . Based on a review of research papers presented
at the conference, authors of selected papers were invited to take part in
the interviews. The selection of papers was limited to works that reported
on corpus-based linguistic analyses with a non-automatized setup andtook a strong methodological focus. Studies tackling complex linguistic
subjects were prioritized over studies with more simplistic targets (e.g. word
frequency comparisons).
The participants of the second series of interviews (I2) were selectedamong the extended group of highly expert corpus linguists of the German
research community2 . The selection was done based on the number of yearsof corpus experience and visibility within the community.
Overall, 20 interviews were conducted (12 in I1 and eight in I2).

4.2.2 Interviewing method and procedure
The interviews 1 (I1) were set up as open expert interviews driven bya limited number of guiding questions around four topics (see Section 4.2.3).
The interviews were conducted via phone and Skype. They were held inGerman and English and had a duration of 30 to 90 minutes. All interviews
were conducted by the author and the interviewing situation was kept stable
over the dierent interviews.
The interviews 2 (I2) were held as semi-guided expert interviews.
They focused on specic aspects of corpus working practice which were ad1

As held in Birmingham in 2007:
http://www.birmingham.ac.uk/research/activity/corpus/publications/conferencearchives/2007-birmingham.aspx
2
The restriction to the German community was motivated by the higher familiarity
with the community and the possibility to carry out the interviews in everybody's native language, and under the assumption that dierences to other communities would be
negligible.

4.2. Setup of the interviews

95

Figure 4.1: Topics of interview guiding questions
dressed through a set of targeted questions (see Section 4.2.3). Despite using
a more structured interview guideline than in I1, with focused questions, theinterview interaction was not strictly imposed, and the interviewees were
given room to expand beyond the questions, wherever relevant.

4.2.3 Topics and guiding questions
The interviews 1 were investigating on detailed descriptions of the corpus
analysis workow and the respective tasks involved. They enquired on
the usage of tools and requirements related to them. Also characteristics of the users in terms of their professional background and motivation
to use corpora were investigated as well as general comments on corpus
research practice. The guiding questions of the interviews 1 are listed inTable A.1 in Annex A.
Through the analysis of interviews 1 it became clear that theoretical concepts inuence and shape choices regarding workows, tasks and tools use.
Besides this it was observed that analysis methodologies, in terms of workows and tasks, were described with a close linking to the availability andpower of tools and data resources at hand. Thereupon, the objectives of theinvestigation were slightly adjusted for interviews 2. On the one hand, inorder to gain a more open-ended view on the user's requirements for tools
support, the interviews 2 aimed at understanding users actual corpus usage situations (cf. Section 3.3.1) by investigating on high level goals andanalysis objectives and resulting demands regarding tasks and analysis actions related to it, as well as desired characteristics of the corpus
resources including their annotations. On the other hand, in order to understand the fundamental grounding of corpus research practice, theoreticalprinciples and methodological approaches were explicitly enquired on.
The interview guiding questions are provided in Table A.2 in Annex A. In
addition, the interviews 2 were accompanied by a short written questionnaire on user characteristics, which was distributed to the interviewees

96

Chapter 4. Interviews on Corpus Working Procedures

via email before the interview took place (see Table A.3 in Annex A).3
Figure 4.1 lists the core topics of the guiding questions of both interview
series, and shows their mapping from interviews 1 to interviews 2.

4.2.4 Analysis procedure and description
The interviews were rst analyzed separately for each series (I1 and I2).
Afterwards, the results were combined and described along six high-level
topics that emerged from the full panorama of interviews (see below).
For I1, rst the interviews were summarized in written minutes or taperecorded and literally transcribed.4 Second, the interview transcripts wereanalyzed along a set of categories which was built by combining topics addressed in the guiding questions with further topics emerging from the interview data
For I2, the interview data was analyzed according to the qualitative content analysis method by Mayring (2003) (cf. Section 3.2.2.2, and also Helfferich (2005)). The set of analysis categories was initially build from the
topics of the guiding questions, and was continuously rened during theanalysis process. The analysis procedure was divided into three major steps.
First, the interviews were literally transcribed.5 Then the transcripts weremanually processed, assigning each text segment to an analysis category, andthe information for each category was summarized, further structured andinterpreted.
Finally, results from both interview series were merged and are reported
along six high-level categories:(1) User characteristics,(2) Linguistic research objectives,(3) Corpus research procedures,(4) Description and management of corpus data,(5) Tools for corpus analysis, and(6) Corpus methodology
This nal category set, with a ne-grained dierentiation as shown in Figure
4.2 has been derived from the analysis categories of both interviews, whichbuilt on guiding questions amended with additional subtopics emerging fromthe interview data, as explained above.

4.2.5 Scope of results
The results are based on individual views of 20 corpus linguists (cf. Section
4.2.1). They give insight into the perspectives of these experts, which are3

Questions on user characteristics were outsourced to a short questionnaire to limit theinterview duration and to collect general information in a uniform fashion.
4
Due to varying technical circumstances dierent logging procedures had to be applied.
5
Apart from one interview which was summarized from memory due to technical problems during recording.

4.2. Setup of the interviews

97(1) User characteristics
Professional background
Years of corpus experience
Skills of tools usage and programming
Motivation and experiences using corpus-based vs. other approaches(2) Linguistic research objectives
Research subjects(3) Corpus research procedures
Corpus analysis approaches
Workows
Analysis tasks and related analysis actions
Manual work within analyses(4) Description and management of corpus data
Data units of reference
Information types of data
Interlinking of data units and information types and relations(5) Tools for corpus analysis
Expectations towards tools
Characteristics and functionalities
Types of tools
Selection criteria for tools
Specic tool requirements(6) Corpus methodology
Strengths and limits of the corpus approach
Validity critera for corpus studies
Practical challenges and best practices
Figure 4.2: Analysis categories for interview data
based on their individual practical experiences with corpus linguistics, as
opposed to the mainly generic theoretical views expressed in the literature.
Insights gained from the interviews are representative for the interrogated experts and their individual working situations. With regards to theirtimeliness they represent the situation at the time of the investigations6 , but
are assumed to be of relevant signicance yet today (cf. Section 1.3). With
regards to the overall population of corpus linguists they have an informative value but cannot be generalized. Within the entire user-centered studyof this thesis, the interview results were used to inform the design of thequestionnaire survey (cf. Section 4.4). Accordingly, the results to the questionnaire give further indications about the generalizability of the results tothe interview investigations (cf. Section 5.3).
6

The interviews were carried out in 2008 and 2011.

98

Chapter 4. Interviews on Corpus Working Procedures

Table 4.1: Experience and skills of interviewees

Programming/ no
scripting skills yes
4.3

Experience in corpus linguistics
up to 5 years
more than 5 years
4 respondents
4 respondents
2 respondents
10 respondents

Results

The results of both interview series (I1 and I2) are presented together along
the six high-level analysis categories (cf. Figure 4.2). Accordingly, the following sections report on detailed results with respect to user characteristics(Section 4.3.1), linguistic research subjects (Section 4.3.2), corpus research procedures (Section 4.3.3), description and management ofcorpus data (Section 4.3.4), tools for corpus analysis (Section 4.3.5)
and corpus methodology (Section 4.3.6).

4.3.1 User characteristics
The 20 interviewed corpus users possessed between two and up to 20 yearsof corpus research experience, with varying degrees of skills in scripting and programming (see Table 4.1 for their distribution). They haddierent language related professional backgrounds spanning linguistics,
literary studies, philology, translation studies, computational linguistics andcomputer science. The vast majority of the interviewees reported on using
corpus linguistics as their major research method, while having experience with non corpus-based approaches to linguistic studies as well.
Also, it could be observed that the majority of the interviewed experts takepart in a corpus research project within specialized and limited functions.
For example, some experts are mainly involved with corpus building, annotation and related infrastructures, while others are focused on detailed
linguistic analyses on ready made corpora. The role taken in the research team or project typically is linked to the professional backgroundand skill set of the user. For example, dierences relate to the employment
of automatized methods and the extent of manual work that a user invests.
Dierences also concern the depth at which the work is grounded in the
corpus methodological approach or the authority of the corpus as source forinsights. On a surface level the dierent working approaches can also be anindication for how well-acquainted are people with available methods, tools
and analysis options.
Table A.4 in Annex A lists the specic proles of the interviewees withregards to their professional backgrounds, their years of experience in corpus
work, their computational skills and the status of corpus linguistics in their4.3. Results

99

research.

4.3.2 Linguistic research objectives
The research subjects of the interviewees showed a high variation andconcerned dierent levels of linguistic description. Lexico-grammaticalphenomena (e.g. reexive verbs, null subjects, etc.), semantics, pragmatics
and discourse (e.g. metaphors, meanings of modal expressions, etc.), andcollocations and phraseology (e.g. xedness of expressions, verb phrases,etc.), as well as the study of (non-standard) language varieties, languagechange and rst and foreign language acquisition.
The majority of interviewees7 reported on studies about variation inlanguage phenomena or some kind of comparative analysis with regard to time (diachronic), register/genre, language varieties, or dierent languages (including native language vs. learner language). A smaller numberof interviewees8 focussed their work on technical and infrastructure aspects of corpus linguistics such as methodologies for corpus annotation andmanagement or tools design and use.
Table 4.2 gives an overview of the dierent research subjects tackled by
the interviewees. Most of the interviewees are working on more than one of
the listed subjects.

4.3.3 Corpus research procedures
4.3.3.1 Overall analysis approaches
Regarding the overall analysis approach of corpus studies, as found in
the state-of-the-art (cf. Section 2.4.2), the interviews attested that corpora
are either approached with the analytical aim of testing a specic research
hypothesis or they are employed with the aim to carry out any kind of
exploratory analysis. In the former case, linguistic theory is the starting
point to inform corpus investigations, while in the latter case the corpus
data serves as starting point to build theory. Accordingly, the setup of thedierent studies that interviewees reported on mainly diered in terms of
how qualitative and quantitative methods were used, and in which
order they were applied (see Section 4.3.3.2). Either qualitative aspects ofthe data are used to drive a quantitative corpus analysis (i.e. usually inthe case of hypothesis-driven analyses), or a quantitative corpus analysis isused to inform the selection of data for an in-depth qualitative analysis (i.e.
usually in the case of an exploratory pre-study).
The interviews indicated that hypothesis-driven analyses are theprevalent approach to studying corpus data and are well dened in terms
78

17 out of 20
3 out of 20

100

Chapter 4. Interviews on Corpus Working Procedures

Table 4.2: Research subjects addressed by interviewed corpus users
Linguistic research subjects− discourse analysis, media discourse, collocations in media discourse− word uses (metaphorical, literal) in political discourse− metaphor analysis− register analysis− corpora in translation teaching− language variation− language change, diachronic historical grammar analysis, synchronousand diachronous analyses of morphosyntax of Slavic languages− English varieties, verb phrases in English varieties− noun collocations, verb-noun collocations, syntagmatic xedness ofmultiword expressions− study of negation− grammar, lexicography− word formation− construction grammar− formal syntax-semantic interface− pragmatic and syntactic generalizations− learner language, second language acquisition− semantic annotation tools− corpus linguistics for use in NLP tools− corpus creation and annotation for non-standard varietiesof their procedure (see Section 4.3.3.2). Exploratory analyses are lessformally established in terms of their structure and procedure. Often exploratory analysis phases blend into hypothesis-driven analysis processes. The relevance of exploration is considered high, however within
analysis procedures and workows exploration does not have a dened place
among the main steps of pre-structured analyses, such as data extraction
and counting (see Section 4.3.3.3 'data exploration' below).
Interviewees described exploratory analyses as dierent ways of corpus
inspection and browsing that are not directed by clearly dened hypotheses
but are rather driven by the data and exploration process itself. Exploration
was described as playing with the data 9 to identify interesting aspects of thedata or to get a holistic view of the data and a general feeling for the data
and annotations. Exploration is also used to get an understanding of the
10variability and extent of a linguistic phenomena in its context , which can
9'I mean you spend hours just playing around with the concordance lines not reallyknowing what you are looking for, until something suddenly crops up, oh that looks interesting! I'll go o and explore that a bit more!' (I1.11)

10'To get to know a phenomena, I have to look at it from all dierent directions, andjust play with the data.'(I2.8) - translation from German by author

4.3. Results

101

expand initial expectations, by looking at it from dierent perspectives. It
is taken as an empirical indication to inform the decision what data to focus
on and what data to disregard for a certain analysis.
Furthermore, reference was made to analyses for the purpose of language teaching, which mainly consist in the search for illustrative ex11 (German: 'Belegsuche' ), and the training of students in corpus
ampleslinguistic methodology, which focusses on the development of annotation
guidelines, calculations of inter-annotator agreement etc. Further insights
into approaches to corpus analysis within the teaching and training context are excluded from the analysis presented in this chapter.

4.3.3.2 Workows
The interviews showed that hypothesis-driven corpus research workows tend to follow similar sequences of processing and analysis tasks acrossdierent studies and users. More precisely, the interviews gave indicationthat most corpus research workows follow a sequence of ve steps.
1. Hypothesis building (denition of the research objective and translating it to a corpus research hypothesis)
2. Corpus acquisition (creation or selection)
3. Data extraction (corpus search)
4. Analysis of extracted data and interpretation
5. Documentation of ndings (reporting and presentation of results)
More coarse-grained the workow can be subdivided into three phases: a preparatory phase (denition of research hypothesis, corpus selec-tion and study plan, steps 1 and 2 above), an analysis phase (data extraction, analysis/processing and interpre-

tation, steps 3 and 4 above), and a concluding phase (reporting and discussion of ndings, step 5

above).

The analysis phase is the core of the corpus study and corresponds tothe corpus analyses documented in the literature (cf. Section 2.4.2). The
analysis phase is in itself a complex task composed of dierent processing
11'We do searches for illustrative examples, for example we check in an aligned parallel

corpus if a pronoun in one language corresponds to a pronoun in the other language or if
it is omitted in the other language.'(I2.1) - translation from German by author

102

Chapter 4. Interviews on Corpus Working Procedures

Figure 4.3: Blueprint of processing steps within the analysis phasesteps which can dier in character and order between dierent studies. Different tasks of the analysis phase are described in more detail in Section
4.3.3.3.
Regarding workows in exploratory analyses, the interviews demonstrated that exploration usually does not follow a dened workow but is typically part of data extraction, analysis and interpretation activities during
the analysis phase. Accordingly, activities for exploratory analysis activities showed to be implemented in shorter circuits and in a more open-ended
and observatory fashion than workows for hypothesis-driven analyses, andthus can be described as 'micro-workows' embedded within single stepsof hypothesis-driven analyses workows. This is very much in line with theoverall purposes of exploration as opposed to hypothesis-testing (see Section'Data exploration' and Table 4.4 below).
Figure 4.3 shows a blueprint of typical steps of the analysis phasein hypothesis-driven studies, based on information collected from the interviews.12
Within workows, two dierent types of transitions were observed between analysis steps: progressive and cyclic transitions. While analysis
steps with progressive transitions are carried out once in a linear fashion,some steps are connected by cyclic transitions which are back-feeding loops
12

Steps only reported for part of the corpus studies are indicated by dashed lines.

4.3. Results

103

that return to the same or a previous analysis step. The following examples
of cyclic analysis steps were reported in the interviews:
 the same analysis is carried out twice or in parallel (e.g. on dierent

subcorpora), so as to compare the results in a subsequent step frequency lists of dierent types (including keyword lists as well as

concordances tables) are used as input/indicator for selecting relevant
items for concordancing

 concordances are re-visited to carry out disambiguation tasks on fre-quency lists

 manually annotated search results are re-imported into the corpus tool
for subsequent searches (with new annotations) [search → annotation→ search on new data]

Figure 4.4 provides a fully progressive workow and a workow containing cycles13 as part of two corpus studies that were reported in the interviews.

4.3.3.3 Analysis phase
Within a typical corpus analysis workow (see Section 4.3.3.2) the analysis
phase builds on the preparatory work of dening a research question, translating it into a corpus research hypothesis and acquiring a corpus which
provides an adequate basis for the research question.

Data extraction and search.
Regarding the extraction of relevant data from a corpus users reported onusing query tools, which are expected to support regular expression andboolean searches, and to allow for querying multi-layer annotations. Being
able to build subcorpora or to lter results by metadata was reported as
relevant within the search process. In addition to using query tools, users
made reference to custom scripts for extracting data, which cannot easily
be targeted with standard query tools (e.g. reduplications of a word, i.e. two
subsequent occurrences of the word, or phenomena, which consist of gaps,
i.e. 'empty subjects'). In fact, powerful query facilities are a key demandfor data extraction tools. The standard and primary data formats which are
extracted from corpora are concordances and frequency lists (cf. also
Section 4.3.4), while users reported to sometimes also go back to the original
source texts.
A main challenge for data retrieval is to nd all relevant occurrences
for a research question (i.e. a heuristics for exhaustive data retrieval), and to
13

The analysis steps printed in italics indicate onsets of cycles.

104

Chapter 4. Interviews on Corpus Working Procedures

Progressive workow: (from interview I1.6)
frequency lists for all nouns (in WordSmith) per subcorpus−→ manual cleaning of lists for non-nouns−→ division of lists into three frequency bands−→ manual selection of three nouns (occurring in all subcorpora)−→ concordancing of nouns with verb in context of 3-5 words (with CQP)−→ export and import of results to ACCESS DB−→ automatic extraction of all verbs−→ import to Excel table−→ manual correction of lemmatization−→ manual assignment of process types (without looking at context of verb)−→ calculation of verb cross tables in WordSmith (left/right context)
Cyclic workow: (from interview I1.5)
pre-study: browsing for dierent pos-patterns in CQP to nd the meaningful
ones (that maximize precision and recall)−→ put patterns into le for batch-search in CQP−→ extraction of two-word patterns with frequencies−→ ranking of results by MI and normalized frequencies with UCS toolkit−→ by cut-o got 100 most meaningful word pairs−→ transformed into le of search strings−→ 'gawk' to get random sample−→ for these concordances are extracted and manually checked−→ for all 100 word pairs POS-sequences with frequencies were extracted−→ signicance test for dierences of sequences (original vs. translation

corpus)Figure 4.4: Examples of a progressive and a cyclic workow

4.3. Results

105

be aware of precision and recall rates of the data extraction process. In
general, interviewees aimed to minimize manual work and maximize automation of the data extraction procedure as much as possible. They reported
that issues concerning precision and recall values and their estimation areoften due to missing annotations or low annotation quality.
The interviewed researchers reported that data extraction often is arepetitive process that includes several cycles of query renements (cf.
Section 4.3.3.2). In fact, a typical approach to improve recall values of results
is to start from (or fall back on) an overly general query for the phenomenonof interest and successively rene it. For example, results to a coarse query
are manually analyzed (by selecting a (random) subset of the results) in order to dene a rened heuristic for searching, which may rely on observed
patterns of false hits or annotation errors. It has been reported that thesepatterns for searching are partly also derived from word or n-gram lists. The
following strategies for dening over general (coarse) searches were named:
searching for word forms or lemmas, searching for typical patterns of deviation that are known, search on strings (i.e. including regular expressions)instead of annotation tags.
Another challenge mentioned was the overall amount of data that data
extraction can yield. While huge-sized corpora can return extensive amountsof data, which are dicult to manage, the search for rare language phenomena can still face data sparsity issues even within large corpora.
When it comes to dening the search targets, interviewees reported
two strategies: (1) to determine search targets based on quantitative indicators (such as e.g. frequency lists, keyword lists, lists of words with high MI
score, factor scores) with the aim to do qualitative analyses on the results,and (2) to determine search targets based on qualitative indicators (such ase.g. words selected based on theory, or based on explorative concordancing
browsing), with the aim to do quantitative analyses on the results.
Dierent entrance points to corpus than search queries were reported
rarely. One interviewee reported on a study that started from an overall
statistical analyses comparing various categories in a corpus to the same
categories in a reference corpus to determine the overuse or underuse of
14
certain categories in relation to the reference corpus.
Also the demand was expressed to have access to full corpus texts in order
to be able to apply custom made scripts for data extraction and analysis.15
Table 4.3 provides a listing of search and extraction activities that werenamed by the interviewees.
14'For these overuse/underuse studies, we feed any corpus category into a statisticalanalysis and use the patterns that emerge to identify potentially interesting parts of thedata.'

15(I2.8)'Sometimes I need access to the entire corpus data that is the text les, to work with

my own perl scripts or perl scripts of colleagues to extract all information I need. (I2.6) -translation from German by author

106

Chapter 4. Interviews on Corpus Working Procedures

Table 4.3: Search-related corpus analysis activities
Data extraction and search∗ dene search query for aproximating a linguistic target structure∗ based on tags (e.g. semantic tags, POS, etc.)∗ POS-patterns (e.g. nouns with verb in context of three/ve words,
phrase structures like NP approximated by POS-sequence)∗ using wildcards∗ lexical search patterns∗ pre-dened groups of words, also based on batch-les with manuallycompiled word lists∗ with stop-word lists∗ for two-word patterns with frequencies∗ for aligned data
∗ extracting sentence positions of most frequent lexical words∗ looking at selected subsets of the data  data reduction∗ successive renement of search patterns (combination of several/subsequent queries)
Data exploration.
The majority of interviewees mentioned some kind of data exploration activity within the corpus analysis process. Exploration activities were reported
as dierent ways of open-ended corpus inspection and browsing.
Interviewees reported that exploration serves to gain inspiration for
new research topics, to deepen the understanding of the data, to serve thepreparation of a specic study, and to ground the scientic validity of a
corpus study. Table 4.4 lists motivations for corpus exploration mentioned
by interviewees, grouped by the aforementioned themes.
In the interviews, in particular activities for understanding the corpus
as a whole were specied more closely, like looking up corpus design cri16
17
teria and annotations , looking up how things get annotated , and looking
18
at a lot of original texts . These exploration activities were motivated by
the aim to know the data and therefore be able to correctly interpret results
and avoid misgeneralization etc.19 Furthermore, users reported on activities
for screening a particular phenomenon by extracting word and string
sequences, inspecting the results and looking at contexts or more generally
16'... one has to get informed about the creation of the corpus and its structure.'- translation from German by author
17'I am repeatedly looking up, how dierent things get annotated.'from German by author
18(I2.4)(I2.7) - translation'I am looking at a lot of texts to get a feeling for what could be relevant and what

wouldn't be' (I2.2) - translation from German by author

19'I think it is common sense that you need to know your corpus to be able to work with

it properly' I2.4 - translation from German by author

4.3. Results

107

Table 4.4: Motives for corpus exploration
Inspiration for research* pre-diagnostics for promising research areas∗ to get ideas for new research topics∗ nd indications for interesting phenomena by starting from the data
Understanding the corpus* to estimate the representativeness and balance of the corpus∗ to estimate the expected error rates in the data and annotations∗ to get a feeling for the entire texts
Preparation of a specic investigation* pre-study to rene proper research aim∗ to discover examples that are not covered by theory∗ to understand the variability and extent of a phenomenon, meaningful characteristics of it, and dependencies between these characteristics(features)∗ observe what is typical, variation of a phenomenon, dierences, patterndetection∗ detect co-occurrences and dependencies among variables∗ to detect new candidates of a phenomenon
Grounding the validity of a study* to help analysis and generalization∗ to help avoiding misunderstandings and short-sighted interpretations;misinterpretation and misgeneralization

108

Chapter 4. Interviews on Corpus Working Procedures

Table 4.5: Exploration-related corpus analysis activities
Data exploration∗ get a qualitative sense of 'what is happening'∗ to make checks and detect possible errors in the corpus∗ nd patterns/repetitive structures around search term; co-occurringwords∗ investigate on specic examples / specic features of the data
∗ compare dierent usages of words (dierent semantic contexts, dierent
grammatical functions / pos-patterns over dierent subcorpora)∗ closer inspection of examples∗ 'close reading' and taking notes, highlighting∗ manually reading through full source texts; view annotations in full
text; reference to source information∗ context/full text inspection for disambiguation, clarication, contextualization∗ visualizing data to facilitate understanding

having a rst walk through query results and observing patterns.20 Users
reported to carry out these activities with the aim to rene their investigation and to be able to dene better search heuristics.21 The exploration
helps to understand better the extent and variability of the phenomenon, itsparticularities and frequency patterns as well as new aspects or instances ofthe phenomenon which are not covered by the existing theory.
Although exploration was named by many interviewees their estimationof the importance of exploration diered. While the majority of users statedthat exploration is very important, it helps to learn a lot of things, andis also relevant for teaching, a small number of respondents indicated that
exploration is rarely relevant for proper linguistic studies, it is mainly for
corpus newbies, and used in teaching as a means to learn, but is less relevant
for experienced corpus users.22
Table 4.5 provides a listing of exploration activities that were named by
the interviewees.
20'When I look at the rst hits then I obviously also look for correlations. I watch out

for particular patterns.'

21

I2.2 - translation from German by author'The rst step usually is an exploratory approach to dene search heuristics that al-

low to retrieve a maximally exhaustive sample of the phenomenon of interest.'

translation from German by author
22(I2.5) -'In teaching we do exploratory analyses. Just checking what is there. And I think manycolleagues who are getting started with corpora do exploration. And after some time, it
becomes more hypothesis-driven work..'(I2.1) - translation from German by author

4.3. Results

109

Classication of results.
The interviews indicated that the closer inspection of extracted data andtheir classication into dierent groups is the major qualitative anlysis step in corpus studies. Classication tasks include the categorizationof examples according to dened criteria, the separation of relevant andirrelevant examples and the labelling with some kind of description. Classication of search results usually requires access to context of the searchhit, in case of semantic classications it can require a wider context or evenaccess to full source texts. Users pointed out that the classication procedure needs to be formalized by annotation rules, which typically build ona good understanding of the phenomenon that is analyzed. In fact, an interviewee stated that building a classication scheme can require to readthrough many results to get an idea of the nature of the results.23 Anotheruser mentioned that classication decisions sometimes can be externalized
to a reference resource (e.g. a classication is done according to informationfrom a dictionary).24 Also classication schemes are often adjusted along
the way of annotating. Users reported on experiences of classifying data by
several people, which require to cross-match annotations and consider theinter annotator agreement.
Overall, classication is reported to be a prevalently manual task,
which can partly be facilitated by sorting operations and searching for certain
patterns in the results. In general, the description of classication procedures
revealed that corpus users are searching for strategies for partly automatingclassication tasks or approximating classication by subqueries.
Mostly, classication activities are done external to the query tool(e.g. by exporting to spreadsheet tools, MMAX, DBs, etc.) and several
interviewees stated the demand to be able to reinsert classied data into
the corpus to be able to continue using the power of query tools for data
extraction. Also, classication of exported data can pose the diculty toinspect a wider context of the data, as the link is broken to the source data.
Doing classications external to the query tool was motivated by a greaterexibility to dene and structure classication labels (e.g. classications
attached to sequences instead of single words, multiple parallel classications,etc.). Table 4.6 provides a listing of classication activities that were namedby the interviewees.
23'One has to read through the hits one by one. [...] and then you start dening anannotation scheme. You have to dene based on what you want to analyze the examples,and how to identify certain analysis categories. Over time this is slowly becoming clear.'(I2.5) - translation from German by author24'[...] for example by delegating annotation decisions to external resources: somethingis considered as idiomatic, if it is listed in the Oxford Dictionary of Idioms.'

translation from German by author(I2.5) -110

Chapter 4. Interviews on Corpus Working Procedures

Table 4.6: Classication-related corpus analysis activities
Classication of results∗ sorting by context or frequency∗ counting of results by group/subcorpus∗ grouping, classication and sub-selection of data, based on linguistic
criteria∗ cleaning results from irrelevant hits
∗ manual classication of individual results∗ data reduction by looking at rened subsets∗ calculation of MI score between dened words∗ descriptive statistics, Chi Square, parametrical tests∗ correcting data and annotations∗ looking at full texts for contextualization, disambiguation and clarication∗ collocation tables with frequencies∗ positive and negative keywords∗ qualitative analysis of factors (feature groups from factor analysis)∗ visualization
Statistical data analysis.
The interviews gave insights into the statistical analyses which are carriedout in corpus studies. Users reported on calculating frequency prolesfor the combined occurrence of dierent information types (e.g. sequences ofstrings, words in specic contexts, POS, syntactic categories, etc.) in the context of a search item, the dispersion of a search item, and overuse/underusestatistics for comparing various features over dierent subcorpora (features
can be specic words, sequences of words, annotations, etc.). Also significance testing was mentioned. The dierent tools used by respondents
included MMAX tools and custom scripts, as well as 'R' and Weka.
Most statistical analyses are done external to the query tools. Thisis related to the fact that the statistical analysis is often preceded by a semiautomatic classication step. Furthermore, the range of statistical functions
available in corpus tools is sometimes to limited.

Interpretation of results.
The interviews showed that for the interpretation step, the dierent analysis dimensions are considered together. In particular, the classicationof phenomena and statistical results are linked back to the occurrenceof these phenomena in their textual context. Also, results are interpretedwith regard to the overall design of the corpus and its annotations.

4.3. Results

111

Table 4.7: Reasons for manual working steps
Results-oriented motivation* analysis is not feasible by automatic means∗ quality of automatic procedure is not sucient/missing reliability
Procedure-oriented motivation* prefered and pleasant mode of working∗ gaining a better understanding of the data
4.3.3.4 Manual work within the analysis phase.
The interviews showed that, despite the use of tools, the manual processing and analysis of corpus data and results makes up a substantial partof all annotation and analysis activities, and thus of corpus studies overall.
In particular for studies in which retrieval tasks cannot be automated(e.g. the search for metaphors) or classications require human understanding (e.g. full text inspection to derive word meanings in their
textual context) manual work is employed. But also automated retrieval
and analysis tasks often require manual post-processing.25 In addition,
corpus users reported that they sometimes prefer the manual analysis
over the automated procedure, because they get a better grasp of thedata and better control of the analysis. Overall, manual work is a current
reality in corpus working practice for dierent reasons, which are related to
the working procedure as well as the working results (see Table 4.7).
Next to naming substantial reasons for the methodological value of manual work, activities of manual work like adding notes and annotations on
text level, also with pen and paper on print-outs, got criticized by several interviewees. The criticism relates to situations where manual working steps
are employed even though the tasks could be automated. These cases are
typically either due to a lack of skills of the users (i.e. a interviewee reported
that she was not aware of batch queries) or due to a lack of support in tools(e.g. support for refeeding annotations into tools).26
Overall, the interviews indicated that corpus users aim at automatingtasks as much as possible, as long as it is not negatively aecting thequality of results. This holds both for data extraction (see above) and fordata processing tasks.
25'I think it is something [...] whose importance is sometimes underestimated. How to

do this in practice? What is been lost during the process? What do I gain from it? Etc.'(I2.4) - translation from German by author
26
For example, one user reported with slight self-mockery: 'And then I simply printed

out the texts and read through them from beginning to end and marked relevant parts witha highlighter ... a method from the stone age...'author

(I1.01) - translation from German by

112

Chapter 4. Interviews on Corpus Working Procedures

4.3.4 Description and management of corpus data
During the interviews corpus users made explicit and implicit reference tothe corpus as empirical data resource. Based on the explanations givenby the corpus users, a set of dimensions for describing the empirical corpus resource could be identied. We distinguish the fundamental textual
units of reference ('markables' ) as raw data that corpora are typicallycomposed into and that we can operate on (e.g. strings, tokens, sequencesof words, etc.) and the linguistic information types ('markers', i.e.,annotations) assigned to them. These linguistic information types include
classications for data units, for example token-level annotations like word,
lemma or part-of-speech, and classications for relations between units, forexample syntactic relations between tokens of particular parts-of-speech. Finally, the dierent presentation formats of corpus data that corpus
linguistics typically operates on, are understood as formalized or establishedcombinations of data units with their linguistic classication, which in parts
are typically aggregated, incorporating frequency, association or position information.
Table 4.8 provides a detailed description of the corpus resource
along the identied dimensions, based on the information gathered fromthe interviews. The last column of the table also gives some examples ofresearch objectives that are typically served by each particular presentation
format.
Among the presentation formats, as highlighted in the Section 4.3.3,
during the analysis phase corpus users are mainly working with the data
presentation formats concordances and frequency lists.27
With respect to the management of corpus data during the analysis
phase, format and encoding conversions as well as the (post-)processing of
linguistic annotations are the most common tasks. Also, interviewees reported on the double-checking the data quality (e.g. source data andannotations), while working with a corpus.

27

A user pointed out that 'I always think of keywords as being a second order derivativeof a frequency list, and a collocation is a second order derivative of a concordance' (I1.2)

Textual
unitsofreference
Corpus /treebank

Full text

Types of tex- Types of relatual units
tions between
units* corpus de- *association
scription
among words andsentences* annotation
guidelines* metadata* comments* aligned corpora- get to know the corpus* metadata entries* quantitative comparisons of
subcorpora* visual presentation of treebanks* descriptive or comparative
statistics on the corpus/corpora* text structure * aligned texts on sentence(cohesion, arg., and word leveletc.)* highlighted text

* comparison of linguistic
units in aligned corpora* syntactic analysis withsparse syntactic annotation* determine meaning of search
item / disambiguation* grammatical structure* statistical infor- * highlighted concordances
mation* direction of con- * concordances with visual an- * syntactic structure / word
text (right or left) notationsorder* co-occurrences of words/lemmas* proof of evidence*
association * frequency lists of n-grams* collocations + frequencies
strengths* variability* ordered lists by some crite- * collocations + context ('colrion
location tables')

113

KWIC
/ * keyword variconcordanc ants
* context size
(narrow/wide)* unit boundaries* annotations
N-gram / * frequency of
collocation occurrence* sizeCorpus presentation for- Examples of related remats
search objectives

4.3. Results

Table 4.8: Characterisation of corpus resource data

114

Chapter 4. Interviews on Corpus Working Procedures

4.3.5 Tools for corpus analysis
The interviews conrmed the central role of various kinds of corpus analysis tools for the corpus linguistics research process. Search and analysis
tools are highly employed for both analysis approaches: hypothesis-driven
as well as exploratory analyses. The interviews gave detailed insights regarding the users' overall expectations towards tools on the one hand, andopen demands and requirements regarding specic characteristics andfunctionalities of tools on the other hand.
Regarding their expected overall support, analysis tools should serve
the retrieval and presentation of the information that is available inthe corpus in a way that fundamentally speeds up the search and analysis
task, and decreases the amount of monotonous manual working steps. They
should make the data easily accessible by providing concordance views
and aggregated information as well as back-linking to context information.
Analysis tools are expected to be able to handle the encoded corpus including possible annotations. Users emphasized that the power of tools
is obviously not only dened by the tools' functionalities but also dependent
on the data and annotations provided.28 Overall, tools are expected to provide exible query facilities that allow for targeted searches and analyses,
as well as adjustable formatting of query results (in terms of the size ofcontext windows, annotation layers, presentation design) for data inspectionand export.29 While generally speaking, tools are seen as instruments whichfacilitate the active engagement with the data in order to derive informationand new insight, tools are also used as means to get going with the corpus
data and for trying out things (pre-study, exploration).30
The interviews demonstrated that within a workow usually a variety
of dierent types of processing and analysis tools are employed forcarrying out dierent tasks. Also, single tasks were approached by meansof dierent tools, e.g. a corpus search was carried out by dening a search
query within a concordancing tool or by writing a search script that operates
directly on the corpus texts. Table 4.9 lists all types of tools mentioned in
the interviews, grouped by the tasks that they were associated with.
The interviews gave indication that transitions between dierent
28'Of course, one can only nd what is there in the corpus. And if, for example, thecorpus is not tagged than a search software is of limited use for searching grammatical(I1.01) - translation from German by author
A user commented that the presentation aspect is sometimes neglected or consideredwith low priority: 'I got the impression that for computational linguists it is always thephenomena.'

29

query that is in the focus of attention, while the display of results is treated secondarily'(I1.4) - translation from German by author
30'I just interrogated them [the items of interest] in lots of dierent ways using all thegeneral corpus things like: positive and negative keywords, ..
available in WordSmith.

just everything that was

And I kept doing it until I began to nd the things that wereinteresting, that I reported on.'

(I1.11)

4.3. Results

115

tools are typically handled in a non formalized fashion. One example aretransitions between concordancer and spreadsheet or statistical software (i.e.
search results are exported for further analysis). Another example is a transition from exported results (e.g. analyzed in a text editor or spreadsheet tool)
back to the full text corpus sources (e.g. accessible through a concordancer
or server-based search tool).
The interviews showed that several factors inuence the user's choice ofa particular tool. The decision criteria relate to general pragmatic aspects,
usability related aspects, and range of the tools functionalities. The following
pragmatic criteria were mentioned: price, time eciency to get the tool towork, availability in work environment, previously known tools, portability,
access from everywhere (i.e., online interfaces). Usability related criteria
included: easy, intuitive and pleasant usage, good documentation with usage
examples, that is the tool should be extremely low barrier to entry andideally working out-of-the-box.31 Research-related functional aspects
include the reliability of the tool, the degree of control for operating it, andthe analytical and search power. Regarding the last point, the interviewees
uttered specic demands for functionalities and features of corpus
tools which relate to corpus management, corpus querying, corpus analyses,
results presentation and export, as well as technical aspects of the tools(see pragmatic criteria). Table 4.10 lists selected user comments regarding
specic requirements related to these aspects.

31'So, ahm, the best tools .. my experience has been, of the past three years, that the best

tools for non-specialists to use, are the ones that are either well accessible, or incredibly
easy to set up.'

(I1.2)

Type of toolconcordancer

Tasks
corpus search

Chapter 4. Interviews on Corpus Working Procedures

Activities
Example software
extraction of concor- e.g. Stand-alone tools like WordSmith
dances, frequency lists
tools, AntConc, TextStat, Wmatrix, etc.;
server-based programs with web interfaces like OpenCWB, XAIRA, BNCweb,
DWDS search based on DDC, Cobuild online, etc.; and tools that come with corpora
like ICECUP III
custom-madecorpus searchcustomizd data extrac- e.g. text processing utilities like grep,
scriptstionbash, gawk, xquery, etc.
statistical tools / n- results analysis
statistical analyses and e.g. SPSS, R, UCS toolkit, weka, SAS, etc.
grams
creation of graphics for
publications
spreadsheet tools manual
processing cleaning and classica- e.g. Excel, ACCESS DB, MySQL etc.
and databases
and data transforma- tion of search results
tions
manual annotation manual
processing adding manual annota- e.g. MMAX, UAM corpus tool, systematictools
and data transforma- tions and tags
encoder, text editor, etc.
tions
visualization tools
display/presentation
supporting data un- e.g. functionalities in Excel, SplitsTree,of ndings
derstanding, creating xslt for rendering etc.
graphics for publications

116

Table 4.9: Types of tools for specic tasks

4.3. Results

117

Table 4.10: Requirements on tools expressed in interviews
Corpus management
architecture that allows for parallel, conicting and overlapping annotations, aligned corpora, multi-modal content (e.g. pictures, audio les)exible management of the corpus les (e.g. drag-and-drop of les, removing/adding data without reindexing, 'on-the-y corpus updating')
handling various character encodings
access to and copying of full text from search tool
calculation of inter-annotator agreement
Corpus access & querying
querying of data with complex annotation layers, aligned corpora andmetadata; full access to all annotation layers
powerful search facilities (e.g. searches with conditions (if-then constructions), searches for reduplications, etc.)
Corpus analyses
analysis of hits in in relation to metadata, sorting and ltering by metadatadelivery of aggregated data (e.g. frequency information)
quantitative analyses on structured annotation layerssorting by structural attributes not just word positions to left/rightcalculation and display of distributional informationinteractive (manual) reordering and grouping of search results
add categories to search results and then include these categories into
the corpus, to be able to address them in new searches
support for (manual) post-processing of query results (classications,labels, comments) and support for re-import of modied data into corpus
support for storing manually processed search results; subsequent analyses on search results (e.g. successive searches and gradual query renement, statistics on manually post-processed results, sorting and subcorpus building of search results, subsetbuilding on results e.g. by metadata
lters or as random samples (e.g. for further manual in-depth analysis))linking to external content (e.g. dictionary look-up), comparison withintegrated reference corpus
support for exploratory analysis approaches (e.g. 'underuse/overuse'statistics)
Data presentation & exportexible adjustment of presentation formats, e.g. context size of KWIC
visual highlighting of data to facilitate analyses (e.g. data annotatedwith a specic description or marked as 'interesting', color coding ofparticular features of the data)
Continued on next page

118

Chapter 4. Interviews on Corpus Working Procedures

Table 4.10  continued from previous pagecombined presentation of dierent related results, e.g. aggregated information and back linking to context information
data export function (e.g. export button instead of copy-paste)
Technical aspects of tools
portability of tools
scalability of tools
interoperability of tools; data conversion, standardization of data formats, query syntax
customer support, tools documentation/help, example-based help functionalities for query tools, not just 'formula' of query language
support for storing searches and processing steps

4.3.6 Corpus methodology
4.3.6.1 Theoretical strengths and limits of the corpus approach
Interviewees declare that the major strengths of the corpus linguistics approach lie in its empirical, quantitative and systematic method towards the study of language. Users underline that its empirical grounding
relies on authentic data, often relatively easily acquired in large quantities,
which contrasts with the
The strength of the empirical data-driven approach lies in its openness, which provides a strong exploration and innovation potential. In fact
the authentic data found in corpora oers a more holistic view on a languageor language phenomena than the more conned type of elicited data gained
from designed and controlled experiments. Looking at the data can extend
insights beyond the intuition of a native speaker and thus help to discover'the unexpected', for example by encountering unexpected examples (German: 'Einzelbelege' ) or observing unexpected ways of how a phenomenon
occurs in context. Users also stress that the acquisition of authentic empirical corpus data, even in large quantities, is relatively easy and fast ascompared to gathering data in eld studies or controlled experiments.
The quantitative approach to analyzing the data allows to detect patterns, regularities and dependencies between factors by means of quantication, for example by observing unexpected frequency relations, and to generalize ndings over a (large) set of examples. As compared to binary grammaticality judgements collected from native speakers, quantitative methods
applied to large amounts of data allow to identify and measure the degree ofprototypicalness or systematic of a structure without the need to conclude
on categorical judgments of grammaticality.

4.3. Results

119

The systematic approach to analysis guarantees reliable and replicable
results. According to the interviewees, working with concrete data, positively
enforces the explication of methodologies, i.e., through formulating concrete
hypotheses (to be tested or rened), and by following a systematic procedure
for all analysis steps and their documentation.
In order to exploit the described potential of an empirical approach,tools should provide extended support for data-driven analyses andexploration tasks, e.g. by highlighting patterns and the data distribution
as well as by means of linking quantitative data to examples they are derived
from.Also support for quantitative approaches to data analysis should
be provided by tools. The strengths of a systematic methodology, calls forsupport to facilitate or enforce the documentation of data and analysis workows and thus the traceable formalization of research methodology.
The interviewees comment that the main theoretical limit of corpus
linguistics concerns its restricted applicability to research on language inuse, as opposed to studying the cognitive system underlying language competence. This implies that any corpus user needs to be well aware of thenature and characteristics of the corpus in order to estimate its suitability
for the scope of a planned study. Only if a corpus provides a valid basis fora certain study, research results can be meaningful also for a wider context.
The question of generalizability, however is a challenge in its own and isaddressed in Section 4.3.6.3 below.

4.3.6.2 Validity
Validity criteria relate to the theoretical strengths of the corpus linguistics approach. First, in relation to the empirical and quantitative
method they stressed that the availability of adequate and reliable
corpus resources (i.e. adequate size of corpora and balancing of texts,
availability and quality of annotations) is a crucial condition for a valid
analysis. In particular, the validity depends on the representativeness ofthe corpus and related to it the generalizability of derived ndings. The
importance of the corpus as research basis leads to the need for support
in getting to know the corpus, to learn about its content and annotations and enable the user to evaluate its representativeness for a researchquestion under study. Users should have an understanding of the corpus,
starting from information on the sampling as well as corpus metadata32 , andan understanding of entire corpus texts33 . In particular, interviewees alsomentioned the need for setting a gold standard to determine the data
32

An interviewee pointed out that it is important to be aware of the dierence between
selection criteria, the criteria by which corpus texts where chosen, and descriptive criteria,
the criteria later annotated within the texts (I2.4)
33'It can be useful to gain a better picture of the single corpus texts, by starting to readthrough them' (I2.4) - translation from German by author

120

Chapter 4. Interviews on Corpus Working Procedures

quality of the corpus, in particular for certain aspects or subparts of the
corpus that are relevant for the study.34
Second, in relation to the systematic approach users emphasized thatthe careful planning (solid and clearly-dened research hypothesis), execution (systematic operationalization and explication) and documentation of analysis procedures (transparency and replicability, also in resolving
methodological issues) is a prerequisite for the validity of a corpus study.
This also includes formalized approaches to corpus building and annotation.
Regarding the demand for a fully replicable methodology, the importance of the explicitness of any annotation and analysis decision was
stressed. Interviewees claimed that annotations should be marked in the
corpus, where necessary also as multiple parallel annotations or cross annotations. In the same way, analysis procedures should be formalized,
and should be well documented35 , up to the systematic classication36 andpresentation of data. Also, analyses and interpretations should not be done
external to the corpus (in spreadsheets, word docs, etc.), but always enteredright into the corpus to avoid non-reusable and non-automatizable work37
This results in a general demand for policies and best practices.38
Finally, interviewees commented that the overall validity of the ndings also depend on a careful interpretation of the data and an evaluation of the results in relation to the existing knowledge context.39
Both the interpretation of data and their embedding into the scientic context go beyond the data analysis itself, as is often been stressed in the literature (cf. Section 2.4.2.2 and an interviewee's comment 40 ).
34'The global quality of a corpus is often little helpful. You have to investigate about the

local quality for your phenomenon of interest' (I2.8) - translation from German by author

35

A user commented that this is not always easy in real-life workows: 'You might sit

down for days and try to get to the relevant data and suddenly you nd a solution andquickly do the queries, without having it well documented. That is how you might end up
with fragmented solutions.'

36(I1.4) - translated from German by author'You have to create an annotation scheme. You have to state how you want to analyze

the examples, and how do I identify certain analysis categories. You try to dene criteriathat would be reproducible by anybody.'

37(I2.5) - translation from German by author'You should do the classication annotation within the corpus. [...] And nowadays

we have corpus architectures that are supporting these steps, and these architectures arecontinuously improving. (I2.8) - translation from German by author

38

E.g. an interviewee suggested that it should become obligatory for any publishedwork, to hand in data and annotations, as only in this way a work is fully reproducible. 'I
think we have to reach the point, where you have to be explicit about any corpus analysis,
and where you have to submit your data together with any publication to show this is
my interpretation of the data and this is my analysis based on it. Only then an analysis
becomres truly reproducible' (I2.8)

39'The result should be plausible within the context that is already known about thephenomenon based on other analyses.'40(I2.5) - translation from German by author'I see a great potential in corpora for theory building and sometimes I am a bit disap-

pointed to encounter corpus studies that do not include any theory. That do not make an
attempt to create a link to theory building' (I2.6) - translation from German by author

4.3. Results

121Interviewees emphasized that in the process of interpretation the observed quantitative corpus data needs to be related to the theoretical approach under investigation.41 A user pointed out that it can be challenging
to distinguish between the impact of the data (sample) and the impact ofthe validity of a theory.42 Finally, research results derived from corpora
should be cross-validated by other research methods (e.g. psycholinguistic
experiments, native speaker judgements).43
For the interpretation and cross-validation of results, again a detailed and systematic documentation of the initial research assumptions, aquantiable research hypothesis and the analysis steps that lead to the results are needed.

4.3.6.3 Practical challenges and best practices
In relation to the empirical approach, one practical challenge relates to thefact that creating the resources as research basis is work intensive. In
particular the cleaning and categorization of data. As automatic corrections
and annotations are only available and reliable to a limited extent, manual
correction and annotation cycles are often needed.44 As a result of thisintensive workload, suitable big corpora, including reliable reference
corpora are often missing or are hard to nd.45 In addition, copyright
restrictions on the corpora (context limited to snippets) can hinder corpus
work.
Another challenge, relates to handling of resources once they have beenacquired. Indeed, the open-ended nature of authentic data as wellas the often extensive amounts of data are perceived as challenge foranalysis and interpretation, as the authentic data is less controllable thanworking theoretically or with elicited data. In particular, the diculty toexhaustively retrieve all occurrences of a phenomenon in an automatizedfashion (high precision and recall values) are named. On the contrary, users
experienced data sparsity for infrequent phenomena even with large cor41'I don't like to look at things just for the sake of describing frequencies. In fact, I think

results should in the rst place be related to a theoretical approach.'from German by author
42'Often it cannot be distinguished strictly: is this due to the data or is this due to thetheory?'43(I2.4) - translation(I2.4) - translation from German by author'[For the successful completion of a corpus analysis] it should be possible to relate theresults of the corpus analysis to results that have been achieved based on other methods.'(I2.5) - translation from German by author
44'The manual categorization of data for categories that are not part of the existing

annotation ... I think this is something that is needed basically for any topic and is partly
needed extensively.'

45(I2.4) - translation from German by author'There are a few sites that are centrally listing corpora, which is very helpful, but you

would like to have more of these.

[...]of course you don't nd small German corpora

there. I am not sure that there exists any central listing for those.'from German by author(I2.7) - translation

122

Chapter 4. Interviews on Corpus Working Procedurespora. Interviewees suggested to validate the search heuristic by checking
search results against an exhaustive test set which is manually created todetermine precision and recall values. Awareness about precision and recall
values and an understanding of the distribution of results over the corpus are
relevant to decide about the validity of generalization over the results. Thisimplies a demand for tools that would assist the selection of relevant
text sets and the manual correction and reinsertion of these subsets of the
corpus. Also support for relating individual results to the corpus as a whole
are needed (e.g. distribution of results over dierent corpus texts).
While the interviewees emphasized the systematic procedures in corpus linguistics as a strength of the approach, they also reported that inpractice the researcher is often switching back and forth between dierent
analysis approaches and tools, and is rening analyses on the way (cf. e.g.
cyclic analysis phases, Figure 4.4) before arriving to the reportable analysis
result.46 Also, the corpus itself can be evolving during the analysis (e.g. thedenition of annotation categories usually needs to be adjusted dynamicallywhile interacting with the data). This calls for support in carrying out
a systematic and well documented analysis procedure.
Related to manual work, a challenge consists in doing manual processing well. It poses the risk of introducing subjectivity ('the humanngerprint'), which can reect on annotation decisions, classication principles, choice of information during analysis and has to be kept in mind andclearly documented. In general, a demand for raising awareness for theweight of manual processing and including instructions for systematic procedures in manual post-processing into the literature was
highlighted.47
In general, the interviews showed that challenges also relate skills of
the users and approaches used by them. Statistical and quantitative
analyses are considered core of the methodology (as opposed to exemplaryand selected manual analyses). This demands for a clear core skill set
and methodological training of corpus users.
4.4

Summary of results

The interviews result in a detailed overview of corpus research practice based on information provided by 20 corpus linguistics professionals
46'[...] every indicator requires dierent tools and dierent post-processing steps. Only

once you arrive at the statistical analysis, it all starts to come together again, because thenI am dealing with proportions and frequencies of occurrence which can be analyzed in a
relatively homogenous fashion.'

47(I1.4) - translation from German by author'The manual work of post-processing is something that is very often missed out in

corpus linguistics, including in introductory text books.'

by author(I2.4) - translation from German

4.4. Summary of results

123

from dierent educational backgrounds and with varying experiences andskill sets. By discussing key aspects related to workows, tasks, tools andoverall corpus research methodology crucial factors inuencing the needs ofcorpus researchers could be identied and a detailed panorama of activities
within corpus research practice has been collected and spelled out. The gathered insights inform the formulation of open demands for corpus research
procedures and tools (cf. Chapter 6), which are completed by informationfrom the questionnaires. Indeed, the interview results built the groundwork for the questionnaire design by informing the selection of topics,
their structuring into typologies of48 and their instantiation with concrete
examples49 .
The following paragraphs summarize the high level observations fromthe interviews and explain how they informed the questionnaire design. Detailed insights have been described throughout this chapter and the resulting
demands are laid out in Chapter 6.

USER CHARACTERISTICS. Corpus experts come with dierent educational backgrounds and skills. These seem to have a strong impact on
both, the tasks within the corpus workow that a user is focusing on andhow a specic corpus task is carried out. In particular, the professional
background and the level of experience with the corpus-based approach, aswell as the computational skillset seem to be decisive factors regarding theuse of automated vs. manual corpus analyses, and the attention given tomethodological principles and systematic procedures.
To test if these observations can be conrmed for a wider base of corpus
users and in order to more generally analyze preferences and habits in corpus
work in relation to dierent user characteristics, detailed questions on theusers background, education and experience with corpus work were included
into the questionnaire design (cf. Figure B.3 in Annex B and Section 5.3.1).
LINGUISTIC RESEARCH OBJECTIVES. A high variability of research objectives could be observed across corpus users. Their linguistic
subjects are typically specialized and often idiosyncratic in how they are approached. Accordingly, in search of generalizations over dierent studies, it
becomes clear that recurrences in the analysis process are often only foundon higher levels of abstraction. In particular, we identied recurrences in relation to the units of analysis (i.e. lexical level, morpho-syntactic level, text
structure, etc.). At the same, the focus of linguistic corpus studies seems tooften boil down to analyzing and describing the variability of linguistic phenomena within and across corpora (e.g. diachronic, across varieties, acrossgenres, etc.).
These insights were accommodated in the questionnaire design by ex48
49

E.g. a typology for corpus tools allows to group the existing tools by type.
E.g. activities related to doing quantitative analyses on corpus data.

124

Chapter 4. Interviews on Corpus Working Procedures

cluding investigations about specic linguistic research objectives and by
focusing on relevant dimensions of corpus data types and annotations (cf.
Figure B.17 in Annex B and Section 5.3.5) as well as analysis activities fororganizing/processing and inspecting extracted corpus examples instead (cf.
Figure B.9 and B.11 in Annex B and Section 5.3.4).

CORPUS RESEARCH PROCEDURES. Most corpus studies seem tofollow a classic hypothesis-driven design with ve steps: a preparatory phasefor hypothesis denition and corpus building, an analysis phase for data
extraction, analysis and interpretation, and a concluding phase for the reporting of ndings. It was observed that the prominence of dierent analysis
steps varies across studies and users. For example, a study setup can include
the corpus building step or can start from an existing corpus. Also, data extraction and analysis steps are often iterated within one study. In addition,
many users indicated that corpus exploration tasks are relevant, while theyrarely have a dened place in the analysis workow.
Based on these results, the questionnaire investigated to which extent
each step within a classic corpus study applies to the experiences of dierent
users (cf. Figure B.6 to B.15 in Annex B and Section 5.3.4). In particular
it zoomed into the tasks and activities related to the phases of analysis
and interpretation, and also into exploration tasks. The answer options
for activities related to each task were informed by the information collectedfrom the interviews. Related to corpus exploration, both a separate question
on the aims behind exploration (cf. Figure B.7) and an analysis step for'getting to know the corpus' (cf. Figure B.6) was included, as activities forgetting familiar with a corpus, its content and annotations were recurrently
named by the interviewees as examples for corpus exploration.
DESCRIPTION AND MANAGEMENT OF CORPUS DATA. Theinterviews showed that corpus users are referring to the corpus resource indierent ways. Based on the collection of the dierent mentions, we dened aset of dimensions for describing corpus data. On the one hand, these include
the dierent established presentation formats that users deal with. On theother hand, these include the fundamental textual units and the linguisticclassications assigned to the units and to relations between the units, on
which the presentation formats are built. We collected lists of examples foreach dimension and showed how the three dimensions relate to each other(textual unit, linguistic classication and presentation format), and how theylink to examples of research objectives.
Building on this information, questions to investigate on the relevanceof data formats and annotations were included in the questionnaire, as wellas on the relevance of the display of dierent data formats in parallel50 (cf.
Figure B.17 and B.18 in Annex B and Section 5.3.5). Questions on corpus
50

Several interviewees had reported to switch between multiple presentation formats.

4.4. Summary of results

125

management were not included, as they were mainly technical in nature.

TOOLS FOR CORPUS ANALYSIS. The interviews conrmed the central role of dierent tools in the analysis process and gave detailed insights
into the types of tools used and the users' expectations towards corpus tools.
Based on this data, we dened a typology of tools distinguishing, for example, web-based interfaces from stand-alone tools or custom scripts. Users
also uttered functional demands for tools, which cover aspects related to thesearch in corpora, the handling of annotations, the display and export ofresults, and technical features like support for encoding or tool portability.
The typology of tools was included into the questionnaire to investigate
in detail about the frequency of use of dierent tools. Furthermore, the listof funtional demands informed a question on the importance of dierent toolfeatures (cf. Figure B.4 in Annex B and Section 5.3.2).
CORPUS METHODOLOGY. The interviews showed that corpus users
highly value the empirical, quantitative and systematic nature of the corpus
linguistics approach to language study. Users stressed that the openness ofthe corpus approach comes with a strong potential for exploration, but also
poses challenges in terms of safeguarding the validity of research procedures
and results, for example by starting from a good understanding of the corpus,
by making sure to cover phenomena exhaustively and by explicating theapplied methodology.
These insights are mirrored in the design of the questionnaire in two
ways. On the one hand, we included questions on the motivations for using
the corpus approach, asked about crucial conditions for valid analyses andinvestigated on the frequency of dierent reporting activities (cf. Figure B.5
and B.15 in Annex B and Section 5.3.3 and 5.3.4). On the other hand, we
included a number of questions on the potential of visualizations (cf. Figure
B.19 and B.19 in Annex B and Section 5.3.6) as an innovative way to further
support the exploration and (open-ended) analysis of corpus data.
To conclude, the interviews allowed to gain a broad overview of corpus research practice and a contextualized in-depth understanding of corpus analysis processes. The interviews further served to collect a wide variety ofexamples for all aspects addressed, ranging from dierent tools and tool
functionalities, analysis activities, annotation types and data formats, overtheoretical principles, to methodological and technical challenges.
Furthermore, the interviews built the groundwork for the questionnaire
design. The results guided the selection (and exclusion) of topics and related
questions in line with the overall research objective to identify user requirements towards corpus tools for linguistic analyses. The interview results
also informed the questionnaire design by delivering viable sets of answer
options for most questions. Last but not least, the holistic and contextualized insights from the interviews helped the interpretation of the quantitative

126

Chapter 4. Interviews on Corpus Working Procedures

results of the questionnaire, by facilitating the identication of interdependencies between results on dierent aspects of the corpus research procedure.
Results of the interviews together with results from the questionnaire
survey are the source for the description of open demands for corpus
research and tools in Chapter 6.

-Chapter5-

Study on the usage of corpora for linguistic
analyses - questionnaire

127

128

Chapter 5. Questionnaire on Corpus Working Procedures5.1. Aim of the questionnaire survey

129

Aim of this chapter

This chapter aims at presenting the last step of the user-centered investigation of corpus linguistics research practice, which is implemented
by a large-scale questionnaire survey. It describes the design and setupof the survey and reports on its analysis procedure and results.
The chapter starts with the aim of the questionnaire survey (Section
5.1). It continues by detailing its setup (Section 5.2), including the target
groups, the survey design and the topics of the investigation, as well as theanalysis and reporting procedures. Then it presents a detailed elaboration of
the questionnaire results (Section 5.3) in relation to user characteristics,
Tools, methodologies, tasks and related activities, types of data and visualizations for corpus analysis. The chapter concludes with a short summary of
the results (Section 5.4).
5.1

Aim of the questionnaire survey

The questionnaire survey aims to quantify the dierent aspects of the
corpus research practice, which have been identied and described basedon the interviews, and to relate them to dierent types of users. Whilethe interviews were carried out to collect qualitative data in an open-ended
fashion, the questionnaire survey focusses on a quantitative evaluationof a dened subset of aspects related to corpus research practice (seeSection 5.2.3).
Thus, this structured investigation is designed to assess the relevanceof the addressed aspects of corpus research practice and to determine theirweight and distribution. It also serves to identify relevant additional
aspects, if they should yet be missing. In addition, the survey investigates,
if corpus research practice is implemented in dierent ways by dierent typesof corpus users, and if so it serves to describe the dierences in detail.
5.2

Survey setup

5.2.1 Target group
The questionnaire survey targets corpus linguists who are carrying out
linguistic studies based on corpora. It was promoted through major communication channels of the corpus linguistics community, namely establishedmailing lists1 , as well as through individual reference contacts. The approach1

i.e. corpora list, mailliste für Gesprächsforschung, linguistlist, etc.

130

Chapter 5. Questionnaire on Corpus Working Procedures

aimed at maximizing the distribution of the survey to a wide variety of corpus linguists, while reserving the option to exclude contributions out of thetarget group in a second step2 .

5.2.2 Survey design
The survey has been carried out by means of a structured online questionnaire3 . It contains 39 content questions4 , which address relevant aspects of corpus research practice (see Section 5.2.3) as derived from theliterature and the open interview investigations (cf. Chapter 4). The questionnaire presents the topics as closed questions, with predened answer
categories, and open questions that allow for free text answers. Closed
questions make up the major part of the questionnaire (31 out of 39 questions). They serve to request specic feedback on the presented arguments,
partly by ratings on a Likert scale (values 1 to 5, regarding 'importance', 'relevance', 'frequency of use' and 'degree of accordance' ) and partly by meansof single and multiple choice questions (with binary evaluation of 'applicable'
vs 'not applicable' ). In addition, any set of closed questions with evaluation
on a Likert scale is combined with an open question, which allows the participant to add additional or missing information (e.g. 'other' annotation
types). Open questions were used very sparsely to address a small numberof core aspects in an open-ended form.
The survey setup did not enforce that all questions be answered, such thatparticipants could skip single questions and still continue with the questionnaire. Participants were also given the possibility to save their intermediate
results and continue at a later point.
The survey was distributed in English language.

5.2.3 Topics of the investigation
In line with the overall objectives of the user study, the questionnaire survey
addresses relevant aspects of corpus research practice. The detailed
content and overall structure of the questionnaire is primarily informed
by insights from the qualitative interviews (cf. Section 4.4). The detailed
description of individual aspects and their instantiation values are further
complemented by information from the literature (cf. Chapter 2).
In particular, the questionnaire is focused on corpus analysis procedures as opposed to corpus collection and corpus annotation, thus having amore restricted target than the open interview investigations. It addresses
2Single respondents were excluded based on indications of their user characteristics
such as the professional background, e.g. corpus research in bioNLP got excluded
3
Using the Opinio software.
4
Plus additional administrative questions, e.g. the request for consent to the processing
and use of personal data (Q1) and questions about being recontacted, general feedback,etc. (Q41-43).

5.2. Survey setup

131

the core aspects of corpus analysis procedures as identied by the interviews, namely tools, methodology, tasks and data types. In addition,
the survey also collects detailed information on user characteristics inorder to analyze corpus research practices in relation to dierent types ofusers. This is motivated by observations from the interviews which showedthat users with dierent levels of experience and computational skills areconcerned with dierent challenges and objectives. Furthermore, a numberof exploratory questions on attitudes towards visualizations for corpus
analysis are included in the questionnaire. This was motivated by the recent advancements of research on linguistic information visualization (cf.
Section 7.6) and the expectation that visualizations have the potential tobe an additional aid for corpus analyses, in particular in relation to the exploration of large amounts of data. Also, several interviewees indicated theimportance of well-thought presentation formats to support the data analysis
and understanding.
Accordingly, the questionnaire was organized into six thematic units:
1. USER: information on user characteristics
2. TOOLS: usage of tools and their functionalities
3. METHODOLOGY: aims and validity of corpus analyses
4. TASKS: subtasks of corpus studies and related analysis actions
5. TYPES OF DATA: usage of dierent data and annotation types
6. VISUALIZATIONS: visualizations in corpus analysis
Table 5.1 gives an overview of the topics addressed in each section andthe question types used. In Annex B a detailed table with the listing of allquestions is given in Table B.1 and screenshots of the questionnaire as it waspresented to the respondents are shown in Figures B.1 to B.21.

5.2.4 Analysis procedure
The information collected through the questionnaire consists of three typesof data: (1) gradings on a ve-point Likert scale, (2) counts of single andmultiple choice options, and (3) free text answers. For questions with predened answer options, free text elds have been provided for users to enter'other' relevant options that were not listed.
For all questions the response rates are recorded for each item.5 For
grading questions (1) descriptive statistics including mean, standard deviation, median and inter quartile range have been calculated. Table B.9 lists
the descriptive statistics for questions Q11 to Q40. For multiple choice5

The questionnaire conguration allowed participants to skip individual questions.

132

Chapter 5. Questionnaire on Corpus Working Procedures

Table 5.1: Overview of questionnaire structure
TopicsUSER (Q2-Q10)- professional background and activity- experience with corpus work
- use of tools and programming- working languages- age
TOOLS (Q11-Q12)- types of tools- tool features
METHODOLOGY (Q13-15)- motivation for corpus use- validity conditions- focus on linguistic analyses
TASKS (Q16-31)- dening a hypothesis- building corpora- getting to know a corpus- exploring a corpus- manual processing of search results- inspecting search results- quantitative analysis of search results- documenting and reporting ndings- other tasks
TYPES OF DATA (Q32-Q36)- data formats (types)- annotations- parallel display of dierent formats
VISUALIZATION (Q37-Q40)- potential of visualization for corpus
linguistics- use of visualizations for data display
and analysis

Answer types

Single and multiple selectand free text

Likert rating of frequency
and importance and freetext
Likert rating of frequency
and applicability and freetext
Likert rating of frequency
and relevance and freetext

Likert rating of relevanceand select of applicability
and free text
Likert rating of usefulness andfree text

5.2. Survey setup

133

questions (2) absolute frequency counts and percentages of replies havebeen calculated. Multiple choice questions relate entirely to user characteristics and related results are discussed in detail in Section 5.3.1. For open
questions (free text) (3) the answers have been grouped, counted andsummarized. The results are presented in the analyses in the next sections.
Data on a Likert scale is considered as ordinal data in the strict sense,
which means that median and inter quartile range are the most adequate
measures for calculating values of central tendency and dispersion. However,the data gathered by the questionnaire is argued to be considered intervaldata, since the evaluation scales used in the questionnaire dene ranges froma positive to a negative value (e.g., 'very relevant - not relevant'), whichallows for the calculation of mean and standard deviation as basis for the
analysis. Since mean values allow to capture ner grained distinctions than
median values, the results will primarily be discussed with reference tomean and standard deviation, while for reasons of completeness also themedian values and interquartile ranges are provided (see tables as referenced
above) and where relevant reference is made to them.
In addition, results are compared for three dierent subdivisions of the
respondents. Respondents are divided by their user characteristics related
to their professional background, experience with corpus work and programming skills. Detailed information on the subdivision of groups is given in
Section 5.3.1. By means of the Mann-Whitney-U-Test6 , dierences between subgroups are tested for signicance. This non-parametric test isapplicable for independent samples of data on ordinal scale.7 The MannWhitney-U-Test does not assume a normal distribution of the samples. We
apply a two-sided test on the Null hypothesis that the distribution of values from group A and B do not dier in any direction, in the sense thatwith 50% chance a randomly drawn observation of group A is higher thana randomly drawn observation of group B.8 Signicant dierences areattested for p-values < 0.5. The direction of the dierences is determinedby comparing the medians.
All calculations have been carried out by 'R' scripts.9 They are complemented by related graphics (see next section) produced with 'R' and Excel.
The results are analysed and discussed in detail for each thematic unit
6

also known as Wilcoxon Rank Sum Test
As discussed above, we can argue for the Likert data to be on interval scale, but
decided to stick to the conservative assumption and consider the data of ordinal scale forsignicance testing.
8
For a strict testing of a location shift of the distributions the two groups comparedhave to be similarly distributed, which is not the case for our data.
9
Excerpts of the most important scripts are listed in Annex D and are provided
as supplementary material together with the dataset and additional scripts under thefollowing url: https://scientificnet-my.sharepoint.com/:f:/r/personal/vlyding_eurac_edu/Documents/Dissertation (Access is provided to the two evaluators and canbe granted to additional persons on request.)
7

134

Chapter 5. Questionnaire on Corpus Working Proceduresof the questionnaire in a separate section (see Sections 5.3 .). The most
relevant trends and signicant results are discussed in the text, while all
results are included in the tables and graphics. Graphics and tables areplaced at the end of each subsection.

5.2.5 Reporting procedure
The analysis of the survey data and related results are reported in a structured way throughout the next sections of this chapter. Each section is
discussing one topic of the study as introduced in Section 5.2.3. Each section starts with a recapture of the survey questions related to the discussed
topic by presenting the question together with its answer type and answer
options.
As explained above, the majority of the survey questions are either questions with predened answer options that are evaluated on a 5-point Likert
scale or open questions that request free text answers. The procedure forreporting on both types of questions is presented here. For Likert scale questions the results are presented as a bar plot10 which combines several answer
options related to one question. Each answer is represented by a horizontal
bar which shows the distribution of Likert scores by all respondents. The
percentages for each Likert score are indicated by dierent color shades of a(horizontally) stacked bar, which is centered on the middle score (value 3 fora 5-point Likert scale). The bars for dierent answer options are displayed indecreasing order of their overall negative values. Figure 5.1 gives an exampleof the plot. For example, for answer option 1, 53
For each set of questions the highest and lowest ranked options are discussed in the text and related mean values are indicated. Also the standard
deviation is indicated in case the answers for a question show an unusuallywide distribution. All descriptive statistics (median, Inter Quartile Range,mean, standard deviation and number of missing answers (NA) for all questions are given in Table B.9 in Annex B. Within each thematic section,
general observations for each question and topic are pointed out and interpretations of the observed data are given.
Finally, for each question a comparison of the results for dierent usergroups is carried out. As explained in detail in Section 5.3.1, the results werecompared in relation to three characteristics of the users: their professional
background, their level of experience and their programming skills. Firstline chart diagrams for each pair of user groups compare the mean values foreach question related to one set of questions, displayed in the same order as
they were presented in the survey. They help to gain a general overview of
the range of mean values and similarities and dierences between the groups.
Figure 5.2 gives an example of a line chart. For answer 3 the mean values of
10

Created in R with the 'likert' package.

5.2. Survey setup

135

the scores assigned by group 1 and group 2 dier remarkably. If dierences
between groups are signicant is calculated in a second step.
For each group comparison a table provides results on signicantly dierent comparisons, as calculated by the Wilcoxon rank sum test (see Section
5.2.4 for details). In addition to the detailed results provided the tables,
observations on their overall meaning are discussed in the text. The tables
indicate the question number and content, the direction of the observed difference (i.e., A < B indicates that A has a lower mean value than B andthus is more important, more frequently used, more applicable ), the W-value
and the p-value (with threshold < 0.05). Table 5.2 shows the comparisonof group 1 and 2 in relation to answer option 3. With a p-value < 0.05 thedierences are signicant for this example. The score for group 2 is smaller
than the score for group 1 which indicates that group 2 has given higher
rates of importance to answer 3 than group 1.
For open questions that required free text answers, results are presented
in a summarized form. Similar answers are grouped together and, if applicable, linked to an abstracted category that is dened to describe the typeof answer. Results on open questions are both presented and discussed in
the text and referenced in tables listing the original answers and/or theirsummarized content.
The questions related to the user characteristics make use of dierent
answer types and their analysis and reporting is discussed together with theresults in Section 5.3.1.

Figure 5.1: Example of a Likert bar plot comparing 4 answers

136

Chapter 5. Questionnaire on Corpus Working Procedures

Figure 5.2: Example line chart comparing mean values for two user groups
Table 5.2: Signicant dierences between groups 1 and 2 in relation to Qx
Q#Qx

Exampletion
answer3

5.3

Results

ques- Dierencegroup2 < group1

W

p-value1725

0.001571

Overall, 268 persons have accessed the questionnaire, of which 111 participants have fully completed the questionnaire, 113 participants have partly
completed it and 44 participants have only opened the start page. Of the 111
completed surveys 3 respondents have been from outside the target group(Biomedical informatics, BioNLP, Sociology) and got excluded. Thus a naldataset with answers of 108 respondents was the basis for all following
analyses.
As high completion rates, a majority of positive mean values of answers, and conrmative responses to open questions are showing, results
are generally supporting the description of corpus work presented in thequestionnaire (see Tables B.9 . and details on open questions in the following sections). Regarding completion rates, out of 25 rating questions
only three questions were skipped by more than 5% of the respondents (Q34
was skipped by 16% and questions Q20 and Q21 were skipped by 6% of
the respondents). Mean values of answers for all questions ranged from

5.3. Results

137

1.46 to 3.52 of conrmation11 , with 68% of the values between 1.46 and 2.49
and only 32% of the values between 2.5 and 3.52. Standard deviation
values range from 0.8 to 1.72. Open questions addressed in the questionnaire received varying response rates, with up to 66.67% of the survey
participants responding to the most responded to open questions. Overall,free text answers did not contradict but add to the descriptions of corpus
analysis practice given in the closed questions in the survey. However, they
did not provide major additions, which indicates that the presented answer
options were mainly complete.

5.3.1 Users
User characteristics, ranging from professional background to programmingskills, were collected via nine questions (Q2-Q10, see Annex B, Table B.2).
Regarding their education and professional background (Q2) users
were allowed to select multiple answers. Overall, 90 respondents indicated alinguistics background, 41 respondents indicated a computational linguistics
and/or computer science background and 39 respondents indicated any otherlanguage related background. The distribution of all answers for Q2 is shownin Figure 5.3.

Figure 5.3: Education and professional background of all 108 respondents,
including multiple indications (Q2)

For the further analysis of the survey data the respondents were grouped
into two disjunctive proles regarding their professional background: LINGUISTICS including linguistics and any other language related subjects
11

1 being the most conrmatory value ('very important / relevant / applicable').

138

Chapter 5. Questionnaire on Corpus Working Procedures

but with no indication of computational linguistics or computer science and
CL/CS including computational linguistics and computer science, partly
combined with linguistics and any other language related subject. With 67
people the group with LINGUISTICS background without computational
formation/skills makes up 62% of the respondents, while the group of users
with computatinal linguistics/computer science background includes 41 people, thus 38% of the respondents. The distribution of the groups including
their subgroups is shown in Figure 5.4.

Figure 5.4: Grouping of users by education and professional background
Professional activities indicated in Q3 ranged from M.A. student overlecturer and researcher to professor, dictionary writer and company director(see Table 5.3 for details).
Table 5.3: Current professional activities of respondents (Q3)
Professional activity
researcher
professor
PhD student
lecturertechnologist
teacher
student
interpreter
lexicographerNN

nb of respondents
41
18
14
14
7
4
4
1
1
4

5.3. Results

139

Regarding corpus experience (Q4, Q5 and Q6), the great majorityof respondents has been working with corpora for more than 4 years (83people, 77%), consults corpora several times a week (66 people, 61%) anduses corpus linguistics as main research methodology (65 people, 60%). Only
one respondent exhibits less than 1 year of experience, 8 people are using
corpora as rarely as several times a year, and 4 respondents marked corpus
linguistics as a methodology that they employ only marginally. The detailed
distribution of indicators for experience with the corpus approach, divided
also by professional background, are displayed in Figure 5.7 (Q4, Q5 and Q6).
For the further analysis, these three indicators were used to divide users intoa group of experienced and a group of less experienced corpus users.12
In addition, all but one respondent are condent with using computers(Q7) and more than 2/3 have (basic) programming skills (Q8). Also the
distinction between users with programming skills (74 respondents) andusers with no programming skills (34 respondents) was used as grouping
for the further analysis.
Accordingly, Figure 5.5 summarizes the three distinct groupings of respondents by professional prole, level of experience and programming skills that were used for further analysis and shows their overall distribution.

Figure 5.5: Overview of three distinctive groupings
Furthermore, results indicated that people work on a wide range of languages with a strong bias towards English, German and other European
languages (Q9, see Table 5.4 for listing). And nally, respondents are welldistributed in terms of age, covering ages between 24 and 70 years, with
about 56% of the users aged up to 40 and 44% aged over 40 (Q10, see Figure
5.6 for the age distribution).
12

The group of 'less experienced' users is dened as: respondents with 'less than 4
years of experience' OR 'less than 8 years' and 'not every week' and 'not main researchmethodology' OR 'less than monthly access' and 'not main research methodology'. It
makes up 71 of 108 respondents. All respondents not falling into these criteria are counted
as 'experienced', which make up 37 of 108 respondents.

Table 5.4: Languages that respondents work with (Q9)
Language family
Germanic languages only

various European languages

languages from everywhere
Romanic languages only
Slavic languages only

Language
English only
German only
English and German
other combinations
total
English and Spanish
English and Turkishother combinations
total
including English
not including English
total
French; Italian; Spanish
Russian; Polish; Circassian

Figure 5.6: Age distribution of respondents (Q10)

Count
15
14
12
5
45
5
2
32
39
15
5
20
3
1

Figure 5.7: Corpus experience in years, frequency of use and status of the approach (Q4, Q5, Q6)

5.3.2 Tools
Information on tools were collected via two questions on types of tools
and tool features (Q11 and Q12, see Annex B Table B.3). Both questions
requested answers on a 5-point Likert scale. For Q11 users had to indicate
how often? they are using tools (1:'very often' to 5:'never'), while for Q12
users had to indicate how important? tool features are for them (1:'very
important' to 5:'not important').
The most frequently used types of tools (Q11) are oce programs,
followed by statistical software and web interfaces with mean values
from 2.19 to 2.78 and standard deviations of 1.35 to 1.43. The least used
tools are locally installed workbenches with a mean value of 3.5 and astandard deviation of 1.57. The usage frequencies of custom scripts fordata extraction and quantitative analysis show the greatest dispersionof user ratings (standard deviation of 1.72 and 1.7). Figure 5.8 provides an
overview of the distribution of ratings for all types of tools. The data isdisplayed from top to bottom in decreasing order of percentages of negativevalues.13
When comparing the usage frequency of tools across dierent user
proles, little dierences were observable on the mean values between experienced and less experienced users, while considerable dierences were observed between users with linguistics vs. computational background and users
with and without programming skills (see Figure 5.10).
In order to check if the distribution of answers from respondents withdierent proles is equal (H0) or not (H1) with regard to the usage of corpus
tools the Wilcoxon rank sum test with continuity correction was applied andsignicant dierences were found for 12 out of the 21 group comparisons (seeTable 5.5).
More experienced users are generally slightly more active with all
types of tools apart from oce programs and statistical software, but
signicant more frequent use could only be ascertained for web interfaces.
Users with a linguistic background as well as users with no programmingskills make signicantly more frequent use of web interfaces and oce
programs than users with a computational background as well as users
with programming skills, who use scripts for data extraction and quantication signicantly more frequently. Also, users with linguistic background make signicantly more use of standalone concordancing tools,
while users with programming skills are signicantly more active in applying
statistical software.
This shows that the technical skills and background have a strong
impact on whether technical and automatized methods are employed while
the level of experience has an overall eect on the frequency of use ofdierent tools.
13

Sum of the 4th and 5th option on the Likert scale.

142

Concerning tools' features and characteristics (Q12, see Figure 5.9),
speed and scalability, exible query language, statistics and exportfunctionalities were rated with highest importance (mean values between
1.6 and 2.04). A feature for initiating ideas were rated least important(mean value 2.81). Together with alternative entries to data it also received some non-responses.14 The features re-import, support for workow and graphic export received the most diverging answers (standard
deviation of 1.16 to 1.30), which however are less dispersed than any answers on the types of tools (Q11).
Dierences concerning the rating of tool features by dierent user proles are marginal (see Figure 5.11). The Wilcoxon rank sum test with continuity correction was applied and signicant dierences were found only in
2 out of 30 group comparisons (see Table 5.5). Signicant dierences withregard to the ease to use and learn the tool were found, both, in relation
to the professional background (linguistic vs. computational ) and in relation
to programming skills. In both cases, for the more technical users the usability of tools is less important as compared to the less technical users. This
indicates that the usability of tools is particularly crucial for less technicalusers.
For the open question regarding further tool features and characteristics 10 out of 108 respondents provided 18 statements of input (Q12).15 Most
mentions concern technical aspects of the tools (8 out of 18 statements)
or comments related to the operation of tools and data management (4out of 18 statements). Only 3 statements relate to concrete functionalities
of tools. All features and characteristics that were named are listed in Table
5.6, together with their frequency of mention, and the feature type.

14

The non-responses might indicate that participants did not understand what wasmeant by the features.
15
Several respondents named multiple demands.

143

Figure 5.8: Usage of dierent types of corpus tools (Q11)

Figure 5.9: Usage of dierent tool features (Q12)144

Figure 5.10: Dierences between subgroups for tools use (Q11)145

Figure 5.11: Dierences between subgroups for tool features (Q12)146

Table 5.5: Signicant dierences between user proles in relation to tools
Q#
Q11_1

Tool types and features Dierence
web interfaces
exp < less exp
CLCS > LING
yes prog > no prog
Q11_2 standalone concordancing CLCS > LING
Q11_3 local query infrastructure
CLCS < LING
Q11_4 scripts for data extraction CLCS < LING
yes prog < no prog
Q11_5 scripts for quantication
CLCS < LING
yes prog < no prog
Q11_6 oce programs
CLCS > LING
yes prog > no prog
Q11_7 statistical software
yes prog < no prog
Q12_10 easy to learn and use
CLCS > LING
yes prog > no prog

W
929
1833.5
855
1936
941.5
461
2202.5475.5
2053
2269.5582
1725
1869.5848

p-value0.009997
0.002575
0.005799
0.0002635
0.004405
1.769e-09
7.588e-11
6.351e-09
1.472e-09
2.061e-09
2.317e-06
0.001571
0.0005172
0.005827

Table 5.6: Additional important tool features and characteristics
Featureplatform independence / cross-platform compatibility
import/export data formate standards
version management/easy to back-up
interoperability between tools
open source or freely available
programmable/scriptablesupport for collaborative workows (change management; tickets/comments; 'linkability'/the ability topoint at 'things'/ address granularity)
combine dierent functionalities of one tool with eachothersupport for non-Roman scripts and diacritics
tokens are visualised as in the real text (case sensitivity;
so upper and lower cases are shown as in the text for
example in the concordance lines and in the word lists)
retrieval of original discourse data (not only aggregates;
KWIC; sample estimates etc.)
if 'cloud-enabled' then airtight and respected with regard to data security / privacy policy / law compliancegood user manual/support

147

Nb
3
2
2
1
2
1
1

Type
technical
technical
technical
technical
availability
operation
operation1

operation11functionality
functionality

1functionality

1

data management
support1

5.3.3 Methodology: aims and validity of corpus analysis
Information on methodological viewpoints was collected via two questionsin the methodology section, on motivations for corpus-based analyses(Q13) and validity criteria (Q14), and one question in the tasks section onaims in exploratory analysis (Q20) (see Annex B, Table B.4). Questions
Q13 and Q20 requested answers on a 5-point Likert scale. For both questions,users had to indicate how much does it apply? to their work (1:'very much'to 5:'not at all'). Question Q14 has been presented as an open question,
which requests free text input from the participants.
The enquiry about general motivations for corpus-based analyses (Q13,
see Figure 5.12) indicates that the retrieval of quantitative information
was rated highest, closely followed by getting ideas and inspiration andthe retrieval of qualitative information (mean values between 1.46 and1.89). The motivation to derive data for building NLP tools was rated
considerably lower (mean value 2.75) and showed the widest dispersion of
ratings (standard deviation of 1.51). This is in line with our expectation, as
the interviews explicitly targeted 'linguists who are doing linguistics studies
based on corpora' and therefore it is expected that only a minority of themis also working on the development of NLP tools.
By comparing the results of dierent user groups (see Figure 5.13) differences are mainly observed between users with dierent professional backgrounds (linguistic vs. computational ) and programming skills in relation
to the motivation to derive data for building NLP tools, and to somedegree for qualitative data in relation to programming skills of the users.
Indeed, the Wilcoxon rank sum test with continuity correction conrmed
signicant dierences regarding 3 out of the 12 comparisons (see Table 5.7).
With regards to the motivation to use data for building NLP tools, significant dierences were found for users with dierent professional background.
Users with a computational background or programming skills showed asignicantly higher motivation to use corpora for building NLP tools. In addition, the retrieval of quantitative information is a signicantly higher
motivation for experienced users than for less experienced users.
Twelve respondents gave additional answers regarding motivations for
corpus use (13 dierent statements), which in fact were all instantiationsof one of the four answer options. Seven statements referred to corpus use
for secondary aims such as teaching, dictionary creation, translation etc,
which, in the wider sense, could be subsumed under the motivation datafor building (NLP) tools, and six statements gave more specic examples of qualitative and quantitative analyses (i.e. linking to the motivations
retrieval of quantitative information and retrieval of qualitative information). All answers in their original wording are given in Table 5.8.
Concerning the aims in exploratory corpus analyses (Q20, see Figure 5.14), detecting patterns was rated highest, followed by understand148

ing phenomena, and details on phenomena (mean values between 1.78
and 2.04). Understanding corpora and gaining ideas/inspiration wererated least applicable (mean values 2.55 and 2.48) and showed the greatest
dispersion of values (standard deviation of 1.09 and 1.16).
The comparison of results across dierent user groups indicated dierences only for users with programming skills vs. users without programmingskills in relation to the aim of understanding the corpus and details on
phenomena (see Figure 5.15). The Wilcoxon rank sum test with continuity
correction conrmed the signicance of dierences for 2 out of 15 comparisons between user groups, both among the user groups with programmingskills and without (see Table 5.9). Understanding corpora is a signicantstronger aim for people with programming skills, while the aim of getting
details on phenomena is a signicant stronger aim for people without
programming knowledge.
Only two respondents provided additional comments related to aims for
exploratory analyses, which were pointing to the 'quantitative validationof qualitatively gained hypotheses' and the 'search for translation equivalents'.
The open question on conditions for a valid and successful corpus
analysis (Q14) received replies by 66.67% of the participants (72 out of 108).
These 72 respondents made 124 dierent statements.16 The answers werelabelled by four categories, relating to resources (55 out of 124 answers),
methodological criteria (36 answers), tools (19 answers) and user skills(14 answers). The three most frequently mentioned conditions refer to theavailability, power and quality of resources and tools. They are followed byconditions that refer to methodological principles, while the least frequently
mentioned criteria refer to user skills. Table 5.10 provides the full list of
conditions and their number of mentions.

Figure 5.12: Motivations for corpus-based analyses (Q13)
16

Several respondents indicated multiple conditions

149

Figure 5.13: Motivations for corpus-based analyses (Q13)150

Table 5.7: Signicant dierences between user proles in relation to themotivations for doing corpus-based analyses (Q13)

Q # ToolDierence
Q13_2 quantitative informa- exp < less exption
Q13_4 gathering data for NLP CLCS < LING
tools
yes prog < no progW
p-value
1052.5 0.03403
625

3.429e-06

1779

0.0001182

Table 5.8: Motivations for corpus-based analyses (Q13)
Replies of interviewees
getting citations for our dictionary entries
Materials development for training future teachers to use corpora
I am creating models from existing data. Corpus
data is mainly used to anticipate user quesries
Use corpora for teaching how to use corpora (forcorpus linguistics classes)
searching and nding translation equivalents
as teaching material so look if a collocation
works or not in one language
teaching skills for corpus-based linguistic analysis to students
do sequential analysis on raw data
Helping assorted others meet their needs; thorugh an analysis of their needs
Comparative study (time-related; genre-relatedetc.)
My main aim is always data. It would be useful to have a tool that facilitated the analysis of
distributional aspects of language.
quality checks on the resource we create
I use BNC while writing in English to nd if aphrase I want to use is correct. A functionality
for collocation as well as colligation retrival
151

Links to:data for (NLP) tools
data for (NLP) tools
data for (NLP) tools
data for (NLP) tools
data for (NLP) tools
data for (NLP) tools
data for (NLP) tools
qual. & quant. analysis
qual. & quant. analysis
qual. & quant. analysis
qual. & quant. analysis
qual. & quant. analysis
qual. & quant. analysis

Figure 5.14: Aims in exploratory analyses (Q20)152

Figure 5.15: Comparison of user groups for aims in exploratory analyses
(Q20)

153

Table 5.9: Signicant dierences between user proles in relation to aims
in exploratory analyses (Q20)

Q # Tool
Q20_3 understanding corpora
Q20_5 details of a phenomenaDierenceW
p-value
yes prog < no prog 1455.5 0.01242
yes prog > no prog 864.5 0.01242

Table 5.10: Conditions for valid and successful corpus analysis (Q14)
Condition
size and quality of corpora, very large corpus
user-friendly, robust, reliable, powerful, exible, and ecient tools, scalability and speed, platform independent
availability of annotated data
document your methods/ replicability/ reproducability
know the data, i.e. content, evaluation of annotations
availability of corpora and metadata
data that ts the study
know the potential of the query tools
good/well dened research hypothesis
competence in analysis methods
format of corpus
researchers skills in linguistics and computation
researchers skills in statistics
Academic Honesty
user space to save searches/analysis steps, search renement
nding (useful) data
query tool with interface to programming language
machine readable annotation
standard formats

154

#
26
16

Type
data
tools

14
12
10
8
7
6
5
4
3
2
2
2
2

data
methodmethoddata
methodskillsmethodskillsdata
skillsskillsmethod
tools

2
1
1
1

data
tools
data
data

5.3.4 Tasks and related activities in corpus analysis
The questionnaire investigated how often users are carrying out the following
eight tasks that typically can be part of a corpus analysis process (seeAnnex B, Table B.5). All questions requested answers on a 5-point Likert
scale. The users had to indicate how often? they are carrying out the task
(1:'very often' to 5:'never').
 Dening a hypothesis (Q16)
 Creating a corpus (Q17) Getting to know the corpus (Q18) Exploring a corpus (Q19)
 Organizing and manually processing examples (Q22) Inspecting extracted data (Q24) Quantitative analysis of extracted data (Q26) Documenting ndings of the corpus study (Q28)

Overall, the responses showed that inspecting data (Q24) is the mostfrequently applied task, followed by the task of documenting ndings(Q28), getting to know a corpus (Q18) and quantitative analyses(Q26) (see Figure 5.16). For the task of getting to know a corpus it
can be observed that the mean value (2.31) is considerably highter that forthe other highly employed tasks (mean values between 1.69 and 2.01) sincea considerably high number of participants (32%) gave a neutral rating (3on Likert scale). To the contrary the task of organizing and manuallyprocessing data (Q22) has a lower mean value (2.06) but is rated lessimportant due to a greater dispersion of the ratings (standard deviationof 1.2) and a respectively higher number of negative evaluations (4 and 5
on Likert scale). The tasks that are applied least frequently are corpus
building (Q17) and corpus exploration (Q19) with mean values 2.41 and2.43 and the greatest dispersion of votes (standard deviation 1.28 and 1.22).
When confronting the results of dierent subgroups of users (see Figure5.17), multiple dierences can be observed for all subgroups, in particular
in relation to corpus building, organizing and manually processing
data and inspecting data. However, only 5 out of the 24 comparisons show
signicant dierences (see Table 5.11). No signicant dierences are foundfor any comparisons between experienced and less experienced users, eventhough experienced users tend to take more time for dening hypotheses, building corpora, organizing and manually processing data anddocumenting ndings than less experienced users. The subgroups of users
155

with and without programming skills show signicant dierences in the frequency of carrying out the tasks of building a corpus (more frequent forusers with programming skills ) and inspecting data (more frequent for users
without programming skills. In addition, but to a degree which is not signicant, users with programming skills show to be more active for tasks related
to getting to know the corpus and less active for corpus exploration
tasks. When comparing users with a primarily linguistic vs computational
background a signicantly more frequent use of manual processing tasks,
data inspection and the documentation of ndings is attested for users
with a linguistic background, while a slightly (not signicant) lower use ofbuilding, getting to know a corpus and and quantifying extracted
data can be observed.
For all tasks that directly relate to the corpus analysis process17 , users
were asked how often? they carry out specic activities related to eachof the ve tasks18 (exploration Q19 → related activities Q21, organizationand manual processing Q22 → related activities Q23, data inspection Q24→ related activities Q25, quantiative analysis Q26 → related activities Q27
and documentation Q28 → related activities Q29) (see Annex B, Table B.6).
Figure 5.18 gives a graphical overview of the mean values of the activities
linked to the ve tasks. Mean values of all responses grouped by task areshown in ranked order of increasing mean value. Lower and upper values
as well as the ranges of mean values (see red bars) dier between tasks.
Documentation activities received the highest ratings with the top score
for the activities selecting and displaying examples (mean value 1.46).
Manual processing activities were rated lowest with the lowest score forthe activity correcting automatic annotations (mean value 3.28).
A comparison of the mean values for the dierent subgroups of users(cf. Section 5.3.1) is provided for all tasks and related activities in Table 5.13.
The range of mean values for specic subgroups of users is wider than therange of values for all respondents (ranging from 1.46 to 3.28), with the mostextreme values being recorded for users with no programming skills (ranging
from 1.18 for the documentation activity displaying examples to 3.84 forthe quantication activity distribution and dispersion of results, see
underlined numbers). By means of the Wilcoxon rank sum test, signicantdierences were detected for 20 out of 99 comparisons among subgroups for
activities related to tasks.19 The values that showed signicant dierences
are also marked in bold in Table 5.13. Details of the signicant dierences
related to activities are given in Table 5.12.
Among activities related to the exploration task (Q19 with activities
17

The tasks of 'hypothesis building', 'corpus creation' and 'getting to know a corpus'
were excluded for this detailed enquiry, as they precondition the analysis process, but are
not part of it.
18
All questions requested answers on a 5-point Likert scale (1:'very often' to 5:'never').
19
In addition ve signicant dierences are attested for tasks, as explained above.

156

Q21, see Figure 5.19), quantitative analyses and exploratory querying
are most frequently employed (mean values of 1.89 and 1.92), while global
statistics for corpus data is the least used activity (mean value of 2.38).
Interestingly the ratings for quantitative analyses and global statistics
dier remarkably, which indicates that the two types of quantitative analyses (locally on the search results vs. globally on the whole corpus) areperceived as properly dierent activities. Only small dierences are foundbetween user groups (see Figure 5.21). Users with programming skills employ
statistics signicantly more often than users without programming skills (seeTable 5.12) and experienced users do more eyeballing of results than lessexperienced users (no signicant dierence).
Usage frequencies for activities related to the manual processing of results dier considerably (Q22 with activities Q23, see Figure 5.20). Resultsof all respondents show that the manual classication, the subquerying on results, the manual grouping and the manual deleting of hits
are the most frequently employed activities (mean values of 2.07 to 2.5),
while manual corrections of annotations, randomization of results
and sorting by annotations are least frequently applied (mean values of
2.84 to 3.28). The subquerying on results sticks out by having the highest
number of middle ratings (Likert rating 2 and 3) and the lowest dispersion
among all manual processing activities (standard deviation of 1.2). The
results dier noticeably between the users with and without programmingskills and users with a linguistic vs. computational background, while thelevel of experience does not make a dierence in the ratings (see Figure 5.22).
The user groups without programming skills or with linguistic background vswith programming skills or with computational background behave similarly
and all manual processing activities (apart from randomizing results)
are more frequently applied by the less computationally skilled users thanby computationally skilled users. Signicant dierences among both subgroups are attested for the manual classication, the manual deletingof hits and the sorting by context words, while users with linguistic vs.
computational background also show signicant dierences for the manualgrouping of results and the grouping by metadata (see Table 5.12).
Among activities for data inspection (Q24 with activities Q25, see
Figure 5.23) the inspection of results one-by-one and the inspectionof sets of results are the most frequently applied activities (mean values
1.89 and 2.09), while users are least frequently referring to additional
information and highlighting parts of results (mean values of 2.45 and2.59). Comparing results across user groups, it can be observed that themore experienced and less computationally oriented users are generally more
active with regards to all data inspection activities (see Figure 5.26).
Signicant dierences between user groups with linguistic vs. computational
background and with vs. without programming skills are attested for theinspection of results one-by-one and for inspecting the context or157

source of results. In both cases the less computationally-skilled users are
more active in carrying out the activities. Users with a linguistic backgroundalso make signicantly more use of highlighting parts of results (seeTable 5.12).
For activities related to the quantication of results (Q26 with activities Q27, see Figure 5.24), the comparison of frequencies is by far
the most applied activity (mean value of 1.66), while the calculation of'keyness' of words and distribution and dispersion are applied least
frequently (mean values of 2.93 and 2.9). Dierences among subgroups ofusers are only found for proles with vs. without programming skills, withoverall higher usage rates by users with programming skills (see Figure 5.27).
Signicantly higher scores are attested for users with programming skills forthe activities of statistical hypothesis testing and the calculation of distribution and dispersion of words (see Table 5.12). We can observe thatfor activities related to the quantication of results the programming skills of
the user have a direct impact on the usage behaviour which is not reected byusers with a computational background unlike observed in most other cases.20
This may indicate that quantication activities which go beyond the (manual) comparison of frequencies are little supported by out-of-the-box corpus
tools but require additional data processing or programming skills. This
intepretation is further supported by the unexpected dierence in ratings
for quantitative analyses vs. global statistics in Q21 (see above), withsignicant dierences for the latter only among subgroups with vs. without
programming skills.
Activities related to the documentation of ndings are the activities
carried out most frequently by all users (Q28 with activities Q29, see Figure5.25). Usage frequencies are similar for all activities (mean values of 1.46 to1.52) with somewhat lower scores for the discussion of the generalizability of results (mean value of 1.77). The comparison of subgroups showsthat experienced users, users with linguistic background or no programmingskills tend to carry out documentation activities slightly more frequently (seeFigure 5.28). Signicant dierences between computationally vs. linguistically trained users and users with vs. without programming skills are attested
for the activities of selecting and displaying illustrative examples (seeTable 5.12).
Finally, an open question investigated about other primary tasks thatare relevant to the participants' corpus work (Q30). The question received responses by only 15 out of the 108 respondents. They partly named or renedtasks that were indeed proposed in the questionnaire, e.g. 'corpus conversion' as part of corpus building or 'own annotations' as part of manual
20

Throughout the questionnaire, the user groups with linguistic background and usergroups with no programming skills as well as computational background programming skills
mostly show similar response patterns.

158

processing. One user highlighted the 'exploration of full texts for retrieving
discourse phenomena', another user mentioned the 'access to audio/video
recordings, aligned with transcripts'. The remaining responses named tasks
related to domains that make use of corpora, but are not strictly speaking
corpus linguistics, namely language teaching, lexicography/dictionary creation, and translation.
Discussion on tasks and activities
When looking at the results for all tasks and related activities (see Figure5.18), we observe that for each task a small number of activities are mostprominent for the task, while the remaining activities are less frequently
performed (see Annex B, Table B.6 for details on the related questions).
This is particularly true for the quantication task (Q26) with the activityof comparing frequencies, for the results inspection task (Q24) withthe the activity of inspecting examples one-by-one and for the taskof manual processing (Q22) with the activity of manually classifying
results.
The results show that in general the more standard activities, which
are either well supported by standard tools or are executed manually arecarried out most frequently (e.g. manually classifying results as opposed
to correcting annotations or sorting by annotations for the manual
processing task; inspecting results one by one as opposed to referring
to additional information' for the results inspection task; or calculate
absolute and relative frequencies as opposed to calculate distribution
and dispersion for quantitative analyses).
Open comments on the activities, which were provided by very few respondents21 , give further context and insights on the results. For the tasksof manually processing results and inspecting data users commented
that several of the named activities are 'not applicable for spoken corpora'or 'are often not possible'. Also related to documenting ndings a userstated that an additional activity consists in 'describing the drawbacks of theused corpus processing tool(s) in describing languages like Turkish'. In bothcases the comments indicate that certain activities might be less frequent
due to missing support in tools etc., but are indeed relevant.
By analyzing the dierences between dierent subgroups of users we observe that the level of experience has an overall smaller eect on the employment of activities than the programming skills or the professional
background of the users. In the following paragraph we will take a closer
look at the dierences within each of the three subgroupings.
When comparing the results of users with a linguistic vs a computational (linguistic) background it becomes apparent that the linguistic
21

Cf. Annex B, Table B.9 . for exact response rates

159

group makes more use of manual and qualitative activities while the computational group makes more use of quantitative and automatized activities.
Accordingly, dierences are particularly big for most activities related to thetasks of 'manual processing' and 'results inspection', but also the selection
and presentation of single examples for 'documentation'. While the computational group assigns overall lower usage rates, the activities indicated withrelatively higher frequency relate to quantication, in particular statistics
in corpus exploration, but also more advanced quantication activities like
distribution and word association measures. This extends even to statistical
hypothesis testing, which would be assumed to be a very corpus linguisticcore activity, and in fact the task of 'dening a hypothesis' is rated higher
by linguists than by computational users (see Figure 5.17). Also, activities
that operate over sets of data, like the randomization of results, are more
frequent among computational users.
For the user groups with programming skills vs without programming skills
the results dier most in activities regarding 'quantitative analyses'. Programming skills greatly increase the use of the various quantication activities, also as part of 'corpus exploration'. Only for the most basic comparisonof frequencies dierences in use are small across the groups. At the sametime, 'manual processing' and 'results inspection' with activities that focus
on single examples or qualitative inspection of results are carried out more
frequently by users without programming skills. This includes also the eyeballing of results for 'corpus exploration'. Activities that require automationor focus on sets of data (e.g. randomization, inspecting sources) are more
employed by users with programming skills.
Overall the subgroups of linguistic vs computational and non-programmingvs programming users show similar usage patterns in relation to quantitative and automatized methods. However, when it comes to quantication activities greater dierences are found in relation to programmingskills while when it comes to detailed manual activities stronger dierences are found in relation to the professional background of users. Thissuggests that users with a linguistic background put a lot of emphasis on
the detailed qualitative analysis of the data, while for the use of advanced
quantitative techniques it seems to be decisive if the user possesses the relevant programming skills, possibly because out-of-the-box solutions are notalways available. Another, probably related, interesting observation is thefact that the use of global statistics is rated considerably lower by people
with limited computational/programming background than the use of quantitative analyses in general, indicating that they indeed are involved withquantication activities, but without necessarily employing global statistics.
Comparing experienced vs less experienced users results show that experienced user spend more time on detailed data inspection activities. In
particular, they eyeball results (for exploration), and they look into thecontext and additional information of the data or compare sets of results.
160

All of these activities require the skill to grasp information from data in its
context and comprehensiveness.

Figure 5.16: Tasks in corpus analysis (Q16 - Q19, Q22, Q24, Q26, Q28)
161

Figure 5.17: Dierences between groups for all tasks (Q16 .)
162

Table 5.11: Signicant dierences between user proles in relation to alltasks (Q16 - Q19, Q22, Q24, Q26, Q28)

Q#
Q18
Q22
Q24

Tool
getting to know the corpus
organizing and manuallyprocessing
inspecting data

Q28

documenting ndingsDierenceW
p-value
yes prog < no prog 1552.5 0.04274
CLCS > LING
1694
0.0316
CLCS > LING
1724.5 0.01267
yes prog > no prog 977.5 0.03745
CLCS > LING
1661
0.01923

Table 5.12: Signicant dierences between user proles in relation to all
activities (Q21, Q23, Q25, Q27, Q29)

Q # Tool
Q21_6 global statistics
Q23_3 sorting by words

Dierence
yes prog < no prog
CLCS > LING
yes prog > no prog
Q23_5 grouping by metadata
CLCS > LING
Q23_6 manual classication
CLCS > LING
yes prog > no prog
Q23_7 manual sorting
CLCS > LING
Q23_8 manual deleting
CLCS > LING
yes prog > no prog
Q25_1 inspecting results one-by-one CLCS > LING
yes prog > no prog
Q25_4 highlighting parts of results CLCS > LING
Q25_5 inspecting context or source CLCS > LING
yes prog > no prog
Q27_4 statistical hypothesis testing yes prog < no prog
Q27_5 distribution and dispersion
yes prog < no prog
Q29_4 selecting illustrative
CLCS > LING
examplesyes prog > no prog
Q29_5 displaying selected examples CLCS > LING
yes prog > no prog

163

W
1480
1702
899
1543
1651.5786
1729.51909.5779
1802.5723.5
1590.51634.5890
1413
1783.5
625
1779
625
1779

p-value
0.007495
0.002611
0.04972
0.025370.001818
0.009936
0.0005591
2.552e-06
0.005474
0.0002256
0.0005424
0.038130.01659
0.03253
0.02962.495e-06
3.429e-06
0.0001182
3.429e-06
0.0001182

Table 5.13: Mean values for tasks and related activities
TASKS and Activities

all

Activities Q29

Activities Q27

Activities Q25

Activities Q23

Activities Q21

DEFINING HYPOTHESES (Q16)
2.28
CORPUS BUILDING (Q17)
2.41
GET-TO-KNOW CORPUS (Q18)
2.31
EXPLORATION (Q19)
2.43
inspecting_sourcetexts
2.18
exploratory_querying
1.92
eyeballing_query_results
2.06
inspecting_details_of_results 2.11
quantitative_analysis
1.89
statistics_for_corpus
2.38
MANUAL PROCESSING (Q22)
2.06
random_subset_of_results
2.94
subquerying_on_results
2.5
sorting_by_context_words
2.47
sorting_by_context_annot
2.84
grouping_by_metadata
2.44
manually_classifying_results 2.07
manually_grouping_results
2.22
deleting_irrelevant_hits
2.25
correcting_automatic_annot 3.28
DATA INSPECTION (Q24)
1.69
inspecting_results_onebyone 1.85
inspecting_sets_of_results
2.09
comparing_sets_of_results
2.11
highlighting_parts_of_results 2.45
inspecting_context_or_source 2.21
additional_corpus_info
2.59
QUANTIFYING RESULTS (Q26)
2.01
compare_frequencies
1.66
word_assoc_cooccurrences
2.46
calculate_keyness
2.93
stats_hypothesis_testing
2.53
distribution_and_dispersion 2.90
graphics_for_analysis
2.36
DOCUMENTATION (Q28)
2.00
documenting_procedures
1.49
describing_the_data
1.52
explaining_ndings
1.52
select_illustrative_examples 1.46
display_selected_examples
1.46
generalizability_of_results
1.77

164

CLCS

LING

yes

no

prog

prog

2.32
2.26

2.18
2.74

2.19

2.59

2.53
2.12
1.93
2.15
2.15
1.78

2.21
2.30
1.91
1.88
2.03
2.12

2.16

2.82

2.03
3.15
2.36

2.51
2.22
2.22
2.46
2.15
2.00
2.18
2.08
1.74
2.10

2.13
2.52
2.37
2.40
2.19
1.87
1.98
2.13
1.98
2.55

2.37

1.88

2.64
2.64

3.13
2.41

2.08
2.84
2.56

2.95

2.18

2.65

2.09

3.03

2.72

2.82

2.21

2.86
2.49

2.79
2.34

2.49

1.81

2.26

1.66

2.72

1.92

2.33

2.00

3.00

1.80

2.49

1.76

3.38

3.22

3.29

3.27

2.00

1.49

1.78

1.47

2.23

1.62

2.06

1.39

2.18
2.13

2.03
2.09

2.74

2.27

2.11
2.14
2.56

2.03
2.03
2.21

2.56

2.00

2.36

1.88

2.74
1.93
1.79
2.39
2.84
2.42
2.84
2.42

2.49
2.06
1.58
2.50
2.98
2.59
2.94
2.32

2.68
1.95
1.61
2.33
2.76

2.39
2.15
1.75
2.75
3.31

2.33

2.97

2.48

3.84

2.28

1.83

1.61
1.63
1.66

1.41
1.45
1.44

2.25
1.99
1.47
1.56
1.58

2.59
2.03
1.52
1.45
1.39

1.73

1.28

1.57

1.21

1.76

1.27

1.58

1.18

1.85

1.72

1.79

1.73

exp

less
exp

2.17
2.30
2.37
2.41
2.13
1.93
1.91
2.06
1.87
2.36
1.94
2.96
2.48
2.40
2.89
2.43
2.01
2.26
2.23
3.29
1.68
1.86
2.03
1.96
2.41
2.06
2.47
1.99
1.58
2.42
2.93
2.57
2.97
2.32
1.91
1.52
1.52
1.49
1.41
1.38
1.72

2.49
2.62
2.22
2.46
2.28
1.91
2.38
2.22
1.94
2.41
2.30
2.91
2.53
2.62
2.74
2.47
2.18
2.15
2.29
3.26
1.70
1.83
2.19
2.39
2.53
2.50
2.81
2.05
1.80
2.54
2.94
2.46
2.77
2.43
2.16
1.41
1.53
1.59
1.56
1.62
1.88

165

Figure 5.18: Mean values for activities related to tasks

Figure 5.19: Exploration activities (Q21)

Figure 5.20: Manual processing activities (Q23)166

Figure 5.21: Exploration activities by user group (Q21)167

Figure 5.22: Manual processing activities by user group (Q23)168

Figure 5.23: Data inspection activities (Q25)

Figure 5.24: Activities for the quantication of results (Q27)

Figure 5.25: Documentation activities (Q29)169

Figure 5.26: Data inspection activities by user group (Q25)170

Figure 5.27: Activities for the quantication of results by user group (Q27)171

Figure 5.28: Documentation activities by user group (Q29)172

5.3.5 Types of data
Requirements in relation to corpus data were collected via questions on thetypes of data presentation formats, the annotation types, the overalland specic relevance of the parallel display of dierent data formats
and open comments on relevant data types (Q32 to Q36, see Annex B,
Table B.7). Questions Q32 to Q35 requested answers on a 5-point Likert
scale, while Q36 gave room for free text answers. For Q32, Q33 and Q35
the users had to indicate how relevant? the proposed data are (1:'very
relevant' to 5:'irrelevant'), while for Q34 the users had to indicate how often?
it is relevant to look at dirent data formats in parallel (1:'very often' to
5:'never'). For Q32 and Q33 users also had to indicate if the presentation
format or annotation is usually available.
Out of eight common presentation formats (Q32, see Figure 5.29), frequency lists of words/strings and lemmas/parts-of-speech are rated
most relevant (mean values of 1.96 and 1.98). Lists of n-grams and theirfrequencies are rated as the least relevant data presentation formats (meanvalue of 2.61). The distribution of ratings for KWIC with wide context
sticks out in that it received the highest percentage of 'very relevant' ratings
and the lowest percentage of 'relevant' ratings and showed the biggest standard deviation among all presentation formats (standard deviation of 1.44).
The strong need for context information by some users is related to theiruser prole as will be discussed further down.
In addition to relevance ratings, respondents also indicted the expectedavailability of each presentation format in Q32. Table 5.14 lists the data
presentation formats in decreasing order of their mean relevance ratings.
It also lists the availability counts22 and the mean ratings for the groupsof respondents that indicated availability and the group of respondents that
did not indicate availability of the respective presentation format. We can,to some degree, observe a correlation between availability and relevance, i.e.
users who indicate that a data format is not available generally mark it lessrelevant. At the same time, a higher number of availability judgements does
not always correlate with higher relevance ratings. For example, the results
for wide KWIC display and narrow KWIC display show that wide
displays are more relevant even though they are less available, which gives
particular weight to their importance.
Comparing the results for the dierent subgroups of users (see Figure5.30), we can observe dierences between all groups, mainly in relation tostring frequencies, wide KWIC, all corpus documents and narrow
KWIC. In addition, dierences between groups with and without program22

The three highest availabilities are marked in bold. Overall the indicated availability
counts seem rather low (e.g. for KWIC displays a higher availability would be expected),
which could be an indicator for an unclear question design and should be kept in mind
when reading the analysis.

173

n-gram frequencies.
For 7 out of 24 comparisons signicant dierences could be identied bymeans of the Wilcoxon rank sum test (see Table 5.15). Between users withand without programming skills signicant dierences concern all corpus
documents and n-gram frequency lists as well as narrow and wide
KWIC displays. While the former two are more relevant for people with
programming skills, the latter two are more relevant for people without programming skills. Also for users with a linguistic background narrow andwide KWIC displays as well as word/string frequencies are signicantly more relevant than for respondents with a computational background.
Full sets of corpus documents are the only data format which is rated
higher by computationally trained users than by users with a linguistic background (non signicant dierence). Between the users with more vs. lessexperience with corpus work no signicant dierences are found, however weobserve that single corpus documents are more relevant for less experienced users, while narrow KWIC displays are less relevant for them (nosignicant dierences). This seems to highlight that in particular users withless experience strongly refer to context information.
Taken together, the results also indicate that users without programmingskills and users with a linguistics background are more focussed on the presentation of the data in context, while users with programming skills andcomputationally trained users also consider the entire document collection
as a relevant data format for analyses.
Out of 18 well-known types of annotations (Q33, see Figure 5.31) only
5 annotations are rated as relevant by more than 55% of the users (ratings
1 and 2), namely part-of-speech, lemma, infos on corpus design, sentence boundaries and metadata on document level (mean values of
1.62 to 2.05). They correspond to the most standard and common corpus
annotations and are also rated as 'usually available' by 32% or more of the
respondents (see Table 5.16)23 . On the lower end of the scale three annotation types are rated as irrelevant/rather irrelevant by more than 48% of
the users, namely discourse information on coherence, discourse information on coreference and pragmatic information (mean values of
3.29 to 3.52), annotations which are hardly available according to user ratings. The ten annotation types with moderate ratings (mean values between
2.55 and 3.10) are annotations of specic linguistic features (e.g. syntactic
information, named entities, etc.), which are commonly known in linguistics,
but are not regularly encoded in corpora. In fact, we can observe a clear correlation between availability ratings and relevance scores for data annotation
types (see Table 5.16 and Figure 5.33).
ming can also be noticed for the relevance of23

Like in Table 5.14 mean ratings of availability are given together with availability
counts and means for ratings with and without the assumption of availability. The threehighest availability counts are marked in bold.

174

When comparing subgroups of users (see Figure 5.32) dierences can beobserved for all subgroups. For 11 out of 54 comparisons, signicant differences can be attested in relation to morphological information, syntactic dependencies and treebank information, discourse coherence,
discourse coreference, named entities, metadata on document level,
metadata on sentence level and sentence boundaries (see Table 5.17).
For all of these annotations apart from both types of metadata users witha computational prole on average give a higher relevance score.
The inclination of the ratings of users with a computational vs. linguistic
background mainly correspond with the ratings of users with and without
programming skills and just dier in their degree. For example sentencelevel metadata are considerably more relevant for users skilled in programming than for computationally trained uses, while sentence boundaries
are considerable less relevant for them when compared with ratings of users
with a computational background. Also, among the groups of users with andwithout programming skills only one signicant dierence can be attested
in relation to syntactic dependency annotations, while for the groups ofcomputationally vs. linguistically trained users signicant dierences can beattested in relation to seven dierent annotation types (cf. Table 5.17).
Overall, information on the corpus design and metadata are rated more
relevant (not signicant) by users with no programming skills or a linguistic
background, which indicates that they aim to get a detailed understandingof the data in its context. At the same time, detailed structural annotations (e.g. in particular on syntactic, semantic or discourse information) areconsistently rated more relevant by users with programming skills.
Also for the more experienced corpus users information on corpus design and metadata are rated higher in relevance (for corpus metadata
the dierences are signicant), and they also consider custom annotations
more relevant. This could be explained by the fact that experienced users are
expected to be more aware about the importance to contextualize their analyses as well as more experience with ne-grained annotations which cannot
be generated automatically. Less experienced users give signicantly higher
relevance ratings to coreference and coherence discourse information.
This may seem surprising but the rating might be due to a somewhat 'naive'
expression of interest for any annotations, while experienced users might havebecome more realistic about what corpora can oer in terms of discourse
information and thus lowered their expectations.
Finally, the relevance of looking at dierent types of corpus data in parallel was addressed (Q34) and the relevance of specic combinations of data
formats was evaluated (Q35). Figure 5.34 shows that users evaluated thegeneral relevance of data formats in parallel very low (mean 2.84). The relevance of concrete parallel data displays as proposed in Q35 was rated slightly
higher, with the most relevant parallel data displays being source texts andannotations and KWIC and annotations (mean values of 2.21 and 2.32)
175

and the least relevant being KWIC and metadata and source texts andmetadata (mean values 2.58 and 2.62) (see Figure 5.35). Overall the ratings for all proposed combinations are rather similar and rather moderate(no strong ratings for specic combinations). Also, the question on parallel
displays (Q35) received the highest number of non responses.24 This seems
to indicate that the concept of parallel data displays is not very prominent
in corpus linguistics practice even though users mentioned in the interviews
that they are often looking at dierent data (cf. Section 4.4, which wasindicated also by a user in the open question (see Q36 below).
When comparing subgroups of users (see Figure 5.36), hardly any dierences are found between users with more or less experience. For respondents
with a linguistic background vs. a computational background, generally the
linguistic users indicate a higher (or partly equal) relevance for combined displays with the exception of source texts and annotations. For KWIC
and metadata and KWIC and source texts the higher relevance ratingsby linguistic users are signicant (see Table 5.18). The biggest dierences
between subgroups of users can be observed among users with and without
programming skills. The users not trained in programming vote all parallel displays more relevant (source texts and annotations are voted
equal), while the dierences are signicant for the following combinations:frequency lists and hits distribution, KWIC and frequency lists
of words/collocations, frequency lists and KWIC or source texts,
KWIC and hits distribution, KWIC and source texts, KWIC andmetadata and frequency lists and collocations. Overall, 9 out of the
30 comparisons show signicant dierences (see Table 5.18).
These results indicate that for people who are approaching the data in aless automatized fashion, data inspection, also by viewing dierent aspectsof the data is considerably more relevant than for people who tend to workwith the data in a less manually or less qualitative fashion.
The open question on any other comments on relevant data types(Q36) received responses from 5 users. The users partly indicated how theywould like to see parallel displays implemented. For example one user
said 'I would prefer simultaneous display over parallel display (windows)',
while another user suggested 'KWIC AND annotations just below the sameline'. It was also pointed out that 'It is dicult to know how 'relevant'such a parallel display is because few tools oer this now.'. The remaining
two answers related to relevant data for socio-linguistic studies (i.e., age andgender markers) and multimodal data (e.g. images, video, audio).

24
About 17% of the respondents skipped this question, while in other parts of thequestionnaire never more than 7% of the participants skipped a question.

176

Figure 5.29: Common data presentation formats (Q32)

Table 5.14:

Relevance ranking of data formats and expected availability(Q32), out of 108 responses

Data format
word_string_frequencies
lemma_POS_frequencies
single_corpus_documents
wide_KWIC
all_corpus_documents
collocation_frequencies
narrow_KWIC
ngram_frequencies

Mean Avail. Mean
count yesavail.
1.96
43
1.57
1.98
38
1.59
2.30
45
1.89
2.30
40
1.68
2.31
31
1.80
2.37
36
1.80
2.43
52
2.10
2.61
23
1.95
177

Mean
no avail.
2.23
2.20
2.60
2.69
2.52
2.65
2.76
2.79

Figure 5.30: Dierences between subgroups for presentation formats (Q32)
178

Table 5.15: Signicant dierences between user proles in relation to therelevance of data presentation formats (Q32)

Q # Tool
Q32_2 all corpus documents
Q32_3 wide KWIC

Dierence
yes prog < no prog
CLCS > LING
yes prog > no prog
Q32_4 narrow KWIC
CLCS > LING
yes prog > no prog
Q32_5 frequency lists of word- CLCS > LING
s/strings
Q32_7 frequency lists of n-grams
yes prog < no prog

W
1521
1805
875
1572
873
1575.5p-value
0.007057
0.0001263
0.02049
0.017230.029620.020751480

0.007776

Figure 5.31: Relevance rating of annotations (Q33)
179

Figure 5.32: Relevance ratings for subgroups on annotation types (Q33)

180

Table 5.16: Relevance ranking of data annotations and expected availability(Q33) (out of 108 responses)

Data format
POS
lemma
corpus_design_info
sentence_boundaries
document_metadata
sentence_level_metadata
morphological_info
section_boundaries
syntactic_dependency_info
lexical_semantic_info
custom_annotation
treebank_info
role_semantic_info
named_entity_tags
chunk_parses_info
coreference_discourse_info
coherence_discourse_info
pragmatic_info

Mean Avail. Mean
count yesavail.
1.62
44
1.40
1.63
49
1.38
1.79
44
1.47
1.94
39
1.63
2.05
35
1.57
2.55
18
1.56
2.56
23
2.22
2.73
23
2.32
2.74
14
2.23
2.84
6
1.40
2.90
4
2.25
2.94
11
2.18
3.01
5
2.00
3.04
9
1.88
3.10
6
2.33
3.29
3
1.67
3.33
2
2.50
3.52
2
2.50

Mean
no avail.
1.78
1.85
2.02
2.12
2.29
2.76
2.65
2.84
2.81
2.91
2.93
3.03
3.05
3.14
3.15
3.34
3.35
3.54

Figure 5.33: Mean values of relevance by availability ratings (Q33)
181

Table 5.17: Signicant dierences between user proles in relation to therelevance of dierent annotations (Q33)

Q#
Q33_3

Tool
advanced morphological information
Q33_4 syntactic dependency
information
Q33_5 treebank information
Q33_9 coherence discourse information
Q33_10 coreference discourse information
Q33_12 named entity tags
Q33_14 metadata on document levelQ33_15 metadata on sentence levelQ33_16 sentence boundaries

Dierence
CLCS < LING

W
915

p-value
0.01212

CLCS < LING
yes prog < no prog
CLCS < LING
exp > less exp

8751518.5932.51408.50.005599
0.019810.01795
0.04822

exp > less exp

1460.5 0.01856

CLCS < LING
exp < less exp
CLCS > LING
CLCS > LING
CLCS < LING

884.5
926
1581
1681
951

Figure 5.34: Relevance rating of data formats in parallel (Q34)182

0.02805
0.029780.047450.0009284
0.03396

Figure 5.35: Relevance rating of dierent data formats in parallel (Q35)

Table 5.18: Signicant dierences between user proles in relation to therelevance of parallel data displays (Q35)

Q#
Q35_3

Tool
KWIC and metadataDierence
CLCS > LING
yes prog > no prog
Q35_5 KWIC and source text
CLCS > LING
yes prog > no prog
Q35_6 KWIC and hits distribu- yes prog > no progtion
Q35_7 KWIC and word/colloca- yes prog > no progtion frequencies
Q35_8 word/string frequencies yes prog > no prog
and collocations
Q35_9 word/string frequencies yes prog > no progand KWIC or source text
Q35_10 word/string frequencies yes prog > no progand hits distribution
183W
1243.5520
1246.5409.5
543.5

p-value
0.01294
0.002863
0.008345
6.345e-05
0.01011459

0.0003558

554

0.003842

588.5

0.01256570

0.01282

Figure 5.36: Relevance ratings for subgroups on data-formats in parallel
(Q35)

184

5.3.6 Visualizations for corpus analysis
In an additional section of the questionnaire, users were asked how they assess the potential of visualizations for corpus linguistics, in particular
in relation to the previously discussed tasks in corpus analysis (see Section
5.3.4) and types of data (see Section 5.3.5). The opinions on visualizations
were collected via four questions on the overall potential of visualizations(Q37), corpus linguistics aspects that can benet most from visualizations (Q38), the use of visualizations for dierent data types (Q39)
and the use of visualizations for dierent tasks (Q40, see Annex B Table B.8). Questions Q37, Q39 and Q40 requested answers on a 5-point Likert
scale, where users had to indicate how useful? they consider visualizations
for dierent application contexts (1:'very useful' to 5:'not at all useful'). Q38
was formulated as open question that requested a free text answer.
Overall, the potential of visualizations (Q38) was evaluated very positively (see Figure 5.37). 85% of the users gave a positive evaluation (Likert value 1 and 2) of the use of visualizations for corpus linguistics (meanvalue of 1.68). The open question on aspects of corpus linguistics thatcan benet most from visualizations (Q39) received 73 statements from 40
respondents.25 The majority of answers (51) related to data aspects. Visualizations of the distribution of hits across the corpus (11 mentions)and of frequencies and statistical information (10 mentions) are thedata aspects that are considered most promising, followed by visualizations
for individual matches and their annotation layers (5 mentions), as
well as visualizations for diachronic data inspection, relations between
multiples variables, association strengths, clusters and comparative
analyses of dierent types (3 mentions each). The remaining answers (18)referred to analysis aspects. Visualizations for the purpose of illustrating
and reporting (7 mentions) and the detection of patterns (5 mentions)
are named most frequently, followed by visualizations for data exploration(3 mentions) and inspiration gaining (2 mentions). Table 5.19 lists all data
and analysis aspects that were brought up by the respondents and indicates
their number of mentions.
Regarding the use of visualizations for dierent data types (Q39,
see Figure 5.38), respondents evaluated the visualization of statistics on
linguistic features, statistics comparing linguistic features and of
the distribution of linguistic items over the corpus as most useful(mean values of 1.53 to 1.62), while a visualization for displaying single
corpus documents was evaluated as least useful (mean value of 2.79).
This corresponds to and conrms the results of Q38.
Comparing the results for the dierent subgroups of users (see Figure5.40), little dierences among subgroups are observed, with the exception of
the visualization of single corpus documents that diers considerably
25

Several respondents made multiple statements.

185

between users with a linguistic vs. computational background, with the computational users being less interested. The Wilcoxon rank sum test conrms
the dierence to be signicant (see Table 5.20).
The results indicated that users are most interested in visualizationsof linguistic items and their statistics across the entire corpus, whilevisualizations of documents or phenomena on document level are consideredless useful. In addition, the results show that the general interest for different data types, as identied in Q32 and Q33 translates to visualization
scenarios, e.g. single linguistic features are more relevant than annotations
and n-grams. At the same time, corpus design information was rated among
the most relevant types of data (Q32), while visualizations for corpus design
and annotation information is considered among the least useful visualizations. This indicates that users might not have a good idea how visualizations
could be of help to inspect corpus design and annotation information.
When looking at the use of visualizations for dierent tasks (Q40,
see Figure 5.39, the results show that visualizations for the documentationof ndings are rated most useful (mean value of 1.61) while visualizations
are considered least useful for the manual processing of examples (meanvalue 2.52). The high use of visualizations for reporting tasks corresponds
to the results of Q38.
Comparing the results for the dierent subgroups of users (see Figure5.41), dierences are mainly observed for the tasks of exploring a corpus
and manually processing examples among users with a computational vs.
linguistic backround and with and without programming skills. In both cases,
the less computational users see visualizations more useful for the tasks. The
Wilcoxon rank sum test indicated that the dierences among user groups are
signicant only for the task of exploring a corpus (see Table 5.20).
Overall, the results of the questionnaire related to visualizations showedthat users are generally positive about the usefulness of visualizations for
corpus work. Interestingly, users evaluate visualizations for those tasks anddata types most useful, for which current corpus tools already oer somevisualizations (e.g. the distribution of information over corpora and texts,
graphics for reporting). This seems to indicate that users have diculties
imagining how visualizations could support a more extended set of tasks
and data types. We also can observe that linguistic users are slightly moreinterested in visualizations than computational users. In particular, for thetask of exploring a corpus the usefulness of visualizations is rated highby users with linguistic background without having clear indications of how
visualizations could support exploration.

186

Figure 5.37: Use of visualizations for corpus analyses (Q37)

Table 5.19: Aspects of corpus linguistics that can most benet from visualizations (Q38)

Data aspects
distribution by metadatafrequencies and statistics
data layers of individual matches
diachronic analyses
relations between multiple variables
clustering
comparison and contrasting data
collocation / association strenghts
semantic relations
syntactic/dependency structures
coreference data
correlations
feature weighting
multivariate analyses
performance analysis
Analysis aspects
illustrating and reporting
pattern detection
data exploration
getting ideas
getting to know corpus data
Non applicable answers
out of context

187# of mentions
11
10
5
3
3
3
3
3
2
3
1
1
1
1
1
# of mentions
7
5
3
2
1
# of mentions
4

Figure 5.38: Use of visualizations for dierent types of data (Q39)

Figure 5.39: Use of visualizations for dierent corpus analysis tasks (Q40)
Table 5.20:

Signicant dierences among groups towards visualizations(Q39+Q40)

Q # Tool
Q39_2 single source documents
Q40_3 exploring a corpusDierence
CLCS > LING
CLCS > LING
yes prog > no prog
188

W
1596
1681
841

p-value0.005309
0.001415
0.02274

Figure 5.40: Usefulness ratings of visualizations for data types by subgroups
(Q39)

189

Figure 5.41: Usefulness ratings of visualizations for analysis tasks by subgroups (Q40)

190

5.4

Summary of results

In the questionnaire survey user preferences, habits and needs related toseveral aspects of corpus analysis practice were investigated. The results
show that corpus experts could relate to the described set of tasks, aims,
data and tools in corpus work. This is indicated by the overall conrmative
responses of users and the replies to open question, which did not contradict
and only slightly completed the depicted corpus reality. The questionnaire
survey allowed to identify the most used and most relevant aspects
in linguistic corpus analyses by means of quantication and to uncoversignicant dierences in ratings between subgroups of corpus users
in relation to their professional background, level of experience and commandof programming.

USERS. The information collected on the user proles showed that thegreat majority of the respondents are rather experienced corpus users (77%have more than 4 years of experience). 83% of the users indicated to havesome linguistic background, while 66% of the users (also) have some programming skills. This indicates that the majority of respondents is very
versatile both in relation to linguistics and to some degree towards a computational understanding. The distinction of respondents into the dierent
user proles (experienced vs. less experienced, linguistic vs. computational,
with vs. without programming ) thus bundles users according to their major
professional strength and focus area, but does not exclude that users haveto some degree overarching interdisciplinary competences.
Finally, it is noteworthy that 82% of the respondents indicated to mainly
work with European languages, while only 5 users reported to solely workwith non-European languages.
TOOLS. User ratings conrmed that the availability and usability of tools
are the major factor for their frequent use. For example, oce programs
and statistical software clearly beat locally installed query infrastructures
or even stand-alone concordancing software. Also, in terms of tool features,the technical quality (speed and scalability) and exibility (powerful querylanguage) are evaluated the most important.
This picture changes somewhat, when analyzing the preferences of specic user proles. Generally, more experienced users prove to be more active
in using various corpus tools. Users with a computational background or
programming skills rely more strongly on scripts for data processing and lesson web interfaces and oce programs. With regards to tool features the usergroups mainly dier on the aspect of usability, where users with a linguistic background and without programming skills judge easy-to-use tools as
signicantly more important. Users provided very little input for innovative
tool features related to advanced search or display functionalities.
191

METHODOLOGY. The retrieval of quantitative and qualitative data andthe gain of of new ideas from the data showed to apply as motivations for
corpus analysis to similar extents, while the motivation of collecting data for
NLP tools did apply considerably less. When comparing results across usergroups signicant dierences were attested in relation to the usage of corpus
data for building NLP tools, an objective that is signicantly more relevant
for users with programming skills and in relation to the usage of quantitative
data which is signicantly more relevant for experienced users than for users
with less experience.
Interestingly qualitative and quantitative information are rated similarly
important by users with and without computational skills/background eventhough the use of tools and the activities related to quantitative analyses are
rated dierently. This seems to suggest that all types of users value the quantitative aspect of corpus work, but are exploiting quantitative informationin dierent ways.
In terms of aims for corpus exploration, the detection of patterns andexploration of phenomena is most relevant for all users, while the comparison of user groups shows that users with programming skills put the same
weight also on understanding corpora, while users without programming are
considerably more interested in understanding linguistic phenomena than inunderstanding the corpus as a whole.
When asked about conditions for a valid and successful analysis users
put a strong focus on technical pre-conditions and quality checks (availability and reliability of data and tools) and less on methodology and procedures.
This, together with our observations on the requirements for tools, seems toindicate that in corpus linguistics there is still primary concern to guarantee
the base conditions for analyses in terms of resources. Accordingly, methodological questions are of secondary concern in a practical working scenario(i.e. as long as a well-annotated corpus is missing, there is no big point todiscuss about conditions for the generalizability of ndings).
TASKS. Among the eight principal tasks in corpus analysis, the inspectionof extracted data is the most frequently employed task across all respondents,
followed by documenting ndings, which actually is not a specic activity of
corpus analyses but part of any research activity. The quantitative analysisof extracted data is rated more frequent than the manual organization andprocessing of results, which refers to qualitative analyses.
Looking into the details of activities applied for each task, the manualclassication of results and the inspection of data one-by-one, as well as quantitative analyses of results (i.e. frequencies) are among the most frequently
applied activities, while advanced statistical analyses, sorting in relation toannotations, access to secondary corpus information (e.g. collocations) areless applied by the majority of users.
When comparing subgroups of users the overall ranking of tasks remains
192mainly the same, though some dierences between the dierent proles areobservable. Users with programming skills compared to users without programming skills indicate a signicantly higher usage frequency for the taskof getting to know the corpus and a noticeable dierence (not signicant)also for building corpora. Users without programming skills are more active in inspecting data and exploring corpora (not signicant). Users witha linguistic background make signicantly higher use of manual processing,inspecting results and documenting ndings, as well as dening linguistic
hypotheses (not signicant) than computationally skilled users.
These observations show that computationally skilled users are more focussed on quantitative approaches and especially users with programmingskills focus on the corpus as a whole, which can be taken as an indicator for
applying automatic approaches for analysis. Users with a linguistic background put a stronger focus on qualitative information, manual analyses
and a systematic theoretical methodology. These trends are further supported by the results related to the specic activities related to the tasks(cf. Section 5.3.4), e.g. a focus on qualitative inspection ('inspecting results
one-by-one') and context for linguistic users ('going back to wider context
and source documents') and more attention to advanced statistical analyses('statistical hypothesis testing' and 'distribution and dispersion of results')
for users with programming skills.
The comparison of users with more or less corpus experience does notshow signicant dierences. Nevertheless, experienced users are more active
in dening hypotheses, building corpora and manually processing examples,
all aspects that require more advanced methodological skills. They are also
carrying out activities more frequently that require skills in grasping datacharacteristics 'intuitively', e.g. comparing sets of results or eyeballing query
results (no signicant dierences).

TYPES OF DATA. The results on data presentation formats and annotations conrm that the availability and the level of familiarity with corpus
data types are the major factors for their frequent use. Indeed the mostwidely available data types (word/string frequencies and narrow KWIC displays) and the most common annotation types (lemma and part-of-speech
information, corpus design information) are rated most relevant.
The results also conrm the focus on context by linguistic users and users
without programming skills (e.g. signicantly higher ratings for 'narrow
and wide KWIC displays' and higher ratings (not signicant) for 'document
and sentence level metadata'), and on automatic data processing by users
with programming skills (e.g. signicantly higher relevance of 'all corpus
documents' and 'n-gram frequencies') and partly for computational users(e.g. higher ratings for 'all corpus documents' (not signicant)).
Finally, both computationally trained users and users with programmingskills show an overall higher interest in annotations (partly signicant), with193

the exception of metadata, which are signicantly more relevant for users
with a linguistic prole and corpus design information, which is more relevant to experienced users and to users without programming skills (no significant dierences). This seems to underline the awareness for methodologicalconcerns regarding the corpus composition by linguistic and more experienced users, while computational users and users with programming skills
are generally more concerned with the processing of corpus data, be it thedierent annotation layers as well as access to all corpus documents.

VISUALIZATIONS. The results related to visualizations show a generalinterest of visual presentation formats in corpus tools, which is particularly
strong for users with a linguistic background and users with no programming skills. The greatest use of visualizations is expected for the tasks of
documenting and reporting ndings, for which graphics are a common meanof communication. In addition, visualizations are also considered useful for
exploration and pattern detection activities, for which currently few adaptedvisualizations exist.
In relation to corpus data types, respondents expect visualizations to be
most useful for frequency/statistical information and distributional information, which corresponds to the data types for which visualizations alreadyare available and known.
At the same time, overall a clear interest in visualizations can be attested,
but users do not seem to have a good idea of the extent of its use for dierent
tasks and data aspects.
To conclude, the questionnaire survey allowed to get a quantitative understanding of the importance and use of dierent aspects of corpus research
practice. It helped to validate the ndings from the interviews with a larger
user base and allowed to identify potentially missing aspects by means ofopen ended questions.
In particular, by means of the questionnaire we could gain a better understanding which aspects of the corpus working practice are particularly relevant or challenging for dierent groups of users with dierent backgrounds
and skills. We could draw a clearer picture of the corpus research practice ofdierent user types. In conrming the major lines of understanding gainedby means of the interviews, the data collected in the user survey is backing
up the results of the interviews further. Both, the data from the interviews,
and the data from the questionnaire survey inform the description of open
demands for corpus research practice and tools support, provided
in Chapter 6.

194

-

Chapter

6-

Open demands for corpus research and tools

195

196

Aim of this chapter

This chapter summarizes the open issues and requirements in corpus
research practice, which were identied in the two user studies (interviews
and questionnaire survey, see Chapters 4 and 5). They are grouped along veaspects of corpus research practice and are presented in the following subsections: (1) corpus resources, (2) general aspects of tools, (3) corpus
analysis procedures with data retrieval, qualitative analyses, quantitative
analyses and interpretation, and (4) methodological best practices.
For each aspect, a short description of the related issues is given andthe resulting demands are presented as numbered lists. References to therelevant sections in Chapter 4 and 5 are indicated for each demand. The listof demands reects the issues and requirements attested by the surveyed
corpus users. Most of the demands are addressed to some degree in current
corpus practice and by current tools. However, all of the demands are notentirely responded to yet, which could be observed by the fact that theypresent recurrent challenges to corpus users.1
In Chapter 7, potential paths to solutions for the listed demands are discussed. It is pointed out, where recent developments already respond to certain demands. Furthermore, the chapter presents a number of suggestions
for potential solutions to the demands at dierent operational levels,
ranging from recommendations for formation, over best practice guidelines,
up to outlines for specic tool features.
6.1

Demands concerning corpus resources

The availability of corpora is one persistent issue in corpus linguistics.
For many languages and genres resources are still sparse or fully lacking, inparticular e.g. for minority languages and certain types of language which
are not genuinely available in written form, such as informal spoken communication. Also, corpora of authentic language face the problem of erroneous
data (e.g., typos or misspellings) and imbalanced data sets2 .
Another challenge related to corpus resources concerns the availability
and quality of annotations. On the one hand, NLP tools for corpus annotation do only cover certain levels of linguistic description and do not reachfull precision. Also, the availability of tools greatly diers between languages(e.g. many languages only oer the most basic NLP tools like part-of-speechtaggers). On the other hand, also manual annotations require a consistent1

Note that the assistance with applying sound methods is one of the formulated demands(Demand 6.4.5), as partly the lack of user skills and awareness results in the expressionof requirements which are already widely addressed by available solutions.
2I.e., news texts are more easy to get hold of than private communications and arethus often overrepresented in corpora.

197

structuring of the resource by sound annotation schemes, which presupposes
the availability of infrastructures for manually annotating corpus data in asustainable way (cf. Section 6.3.2).
Erroneous corpus data or missing and unreliable annotations can leadto follow-up problems for corpus searches like low precision andrecall values. This is decisive for the corpus research process as low recall
can impede statistical analyses, which require exhaustive and reliable result
sets (cf. Section 6.4 for demands regarding theoretical principles and bestpractices).

[6.1.1] Improve centralized documentation of available corpora[6.1.2] Improve support for facilitated corpus creation and annotation[6.1.3] Improve the reliability and coverage of automatic annotations
Demands 6.1.1 to 6.1.3 derive from the users' complaint that suitable, reliable
and large corpora are often missing or are hard to nd (cf. Section 4.3.6.3
and Table 5.10)

6.2

Demands regarding general aspects of tools

Even though much progress has been made on the technical power andusability of corpus tools over time (cf. Section 2.5.3), a number of general
requirements regarding the operation of corpus tools are yet existing. Related
requirements concern the portability and interoperability of tools withregards to dierent operating systems, but also in relation to support fordierent encodings and data formats and their seamless conversion, which
are particularly relevant for transitions between dierent tools. Another setof demands relates to the scalability of tools to large corpora. Finally, theneed for easy to use and easy to learn tools is omnipresent and persistent.

[6.2.1] Improve the portability, interoperability, and scalability of tools[6.2.2] Improve the usability of corpus tools
Demand 6.2.1 was repetitively raised during the interviews (cf. Section 4.3.5
and Table 4.10) and its aspects were rated as the most important tool features in the questionnaire survey (cf. Section 5.3.2 and Table 5.6). The
same holds with regards to demand 6.2.2, with the addition that this demand showed to be signicantly more important for users with a linguistic
background (vs. computational ) and users with limited or no programmingskills (cf. Section 5.3.2 and Figure 5.11).
198

6.3

Demands in corpus analysis procedures

6.3.1 Data retrieval
Demands regarding support for data retrieval concern advanced and exible querying functionalities. Query tools should allow for querying corpora with complex annotation layers and should support regular expression
searches with back-references. This demand for powerful querying facilities
goes together with the abovementioned demand for tools which are easy
to learn and use.
Also, there is a need for support in dening and rening search heuristics for the exhaustive retrieval of data. Issues have been encountered withextracting either too little or too much data. Therefore, functionalities forevaluating the exhaustiveness of retrieved results (e.g. by enabling the
calculation of precision and recall values) are desired. Also, support fordening and rening search queries and keeping track of the often multiple data extraction cycles is required (cf. Section 6.4 for demands regarding
documentation support). In particular, diculties can arise when relevant
annotations are missing or are erroneous and hence search targets have tobe approximated by general search queries.
Finally, users also uttered the demand for enhanced support for display
and export functionalities adapted to corpora with multiple and possibly
concurring annotations.
[6.3.1.1] Provide powerful and exible search functionalities[6.3.1.2] Improve support for nding search heuristics, exhaustive data retrieval[6.3.1.3] Enhance support for display and export functionalities of search
results
Demand 6.3.1.1 was one of the main desiderata of the interviewees related tocorpus query tools (cf. Section 4.3.3.3 'data extraction and search', Section
4.3.5 and Table 4.10) and is backed up by the questionnaire results, whereexible query languages are rated as the second most important tool feature (cf. Section 5.3.2 and Figure 5.9). Demand 6.3.1.2 derives from usercomments on the challenges related to dening search heuristics, and concern
with precision and recall values of results (cf. Section 4.3.3.3 'data extraction
and search', 4.3.5 and 4.3.6.3). Demand 6.3.1.3 was explicitly uttered by interviewees (cf. Section 4.3.5 and Table 4.10, partly in relation to exploration
activities, see Table 4.6) and rated of high importance in the questionnaires(cf. Section 5.3.2 and Table 5.6).

6.3.2 Qualitative analyses, classication and manual work
Demands in qualitative analyses start from exploration tasks of result setswith the aim to better understand the phenomena under analysis and to199

rene the research question. Accordingly there is an open need to facilitate the exploitation of empirical corpus data for gaining inspirationand new linguistic insights by means of support for open-ended data-driven
analyses. Apart from exploration qualitative analyses are mainly related tothe tasks of identifying relevant examples and the analysis of resultsin order to detect patterns and classify data into relevant subsets. In
relation to analysis tools this requires support for the inspection of single occurrences of phenomena in context and the exploration of entire result sets
for a phenomenon. Related to these tasks the demand for adjustable data
displays with dierent views on the data has been expressed (cf. Section
6.3.1).
In a second step, support for manually modifying data and applying
additional annotations (or classication labels) is needed for some corpus
analyses. This includes the classication along pre-dened criteria, the separating of wrong/irrelevant vs. correct/relevant results. In the rst place
this calls for modalities to correct data and apply annotations within
the corpus tool and to re-integrate them into the corpus. Currently,
the process of re-integrating corrected corpus data into readily compiled corpora often poses practical and technical challenges (e.g. time-consuming
re-indexing procedures, technical know-how required to do corpus updates,etc.). In the second place, this also calls for support in building classication schemes, documenting and adjusting them (cf. Section 6.4 fordemands regarding documentation support).
In general, there is a lack in support for carrying out manual processing tasks within corpus tools and for storing and documenting
manual work in a systematic way. When it comes to manual work it
can be observed that it is partly applied due to lacking skills in automating
manual working steps and in part because it is the only feasible way to create quality data. Tools should support the user in using automatic means
where applicable and following manual working steps in a systematic andwell-documented way where manual work is inevitable or favorable. In fact,users have attested that manual processing is sometimes the preferred andmost sound way of working, e.g. for taking a closer look at texts.
Finally, there is a need for quality evaluation of results and for restricting result sets in a meaningful way, if results are too numerous.
[6.3.2.1] Support for open-ended corpus exploration and data-driven analyses[6.3.2.2] Support for the exploration of result sets to detect patterns andoutliers[6.3.2.3] Support for the close inspection of results to identify relevant examples[6.3.2.4] Support for applying (manual) classications and re-integratingthem into the corpus
Demands 6.3.2.1 and 6.3.2.2 are derived from the users principal objec200

tives to gather inspiration, identify new research topics and detect patterns
to prepare and rene analyses by means of exploration as attested by the interviews (cf. Section 4.3.3.3 'data exploration', Section 4.3.6.1, Table 4.4 andTable 4.10). Results of the questionnaire survey show that exploration in general is one of the lesser applied task (cf. Section 5.3.4 and Figure 5.16), whileamong exploration activities quantitative analyses and exploratory querying
are applied most (cf. Figure 5.18). Demand 6.3.2.3 relates to the fact thatthe inspection of data is the main qualitative analysis step in corpus research(cf. Section 4.3.3.3 'classication of results' and Section 5.3.4) with a strong
focus on inspecting examples one by one (cf. Figure 5.18).
Demand 6.3.2.4 derives from the observation that classication tasks arecentral to corpus analyses, but are mainly performed externally to the corpus
query tool for reasons of exibility to dene and apply classication labels(cf. Section 4.3.3.3. 'classication of results'). They are mainly carried out
manually (cf. Section 4.3.3.4). Among the overall set of tasks in corpus
analyses, the manual data processing is among the lesser applied tasks (cf.
Section 5.3.4 and Figure 5.16), while among the manual processing activities
manual classication and manual grouping of results are applied the most(cf. Figure 5.18). Both the inspection of results one-by-one and the manualclassication and manual deletion of search hits are done signicantly more
frequently by users without programming skills and users with a linguistic
background (as compared to a computational background) (cf. Table 5.13).

6.3.3 Quantitative analyses
Demands regarding quantitative analyses concern access to frequency information and statistical signicance testing. Comparative analyses offrequency information for dierent searches, dierent classes of one result
set or comparisons of results across corpora (e.g., in relation to a reference
corpus) are usually carried out on data exported from search tools and could
be facilitated by adapted functionalities for comparisons in the searchtool. Also, frequency proles of the dierent contexts that a linguistic phenomenon occurs in (i.e., WordSketches) as well as overall comparisons ofdierent corpora (e.g., frequencies of certain classes of verbs, certain linguistic constructions) are relevant to corpus users. In addition, information
about the distribution and dispersion of search hits are of interest to
users.
When it comes to statistic hypothesis testing most users make use of specialized statistical software (e.g. 'R' or 'SPSS') and user comments suggestthat it is sensible to use external tools as they are adapted for the purpose
and come with a wide range of functionalities. However, corpus tools could
provide support for preparing the data for statistical analyses or help
understand if specic statistical tests are applicable for the data underanalysis.
201

Finally, tools support to help aggregating and reducing corpus search
results to enable quantication is lacking in current tools.
[6.3.3.1] Support for calculating comparative frequency proles on search
results[6.3.3.2] Support for formalizing and reducing search results for quantication
Demand 6.3.3.1 and 6.3.3.2 relate to the observation that comparisons offrequency data is the most applied quantitative analysis task. It is mainly
carried out external to the query tool, since support within tools is perceived
as limited (cf. Section 4.3.3.3 'statistical data analysis'). The questionnaire
data indicated that quantitative analyses are the main motivation for corpusbased analyses (cf. Figure 5.12 and a frequently applied task (cf. Section
5.3.4 and Figure 5.16), and that it is signicantly more important for experienced corpus users (cf. Table 5.7)). Furthermore, among the activities forquantifying corpus data (cf. Figure 5.18) the comparison of frequencies is
the most applied activity, with statistical hypothesis testing being appliedmuch less.

6.3.4 Interpretation and reporting of ndings
The task of interpreting the data derived from corpora and reporting it
remains to the corpus user and cannot be delegated or automated. It is basedon bringing together all information extracted from the corpora andprocessed during the analysis, and drawing conclusions from it. Finally,
the results of a corpus study and their interpretation is detailed and reported
in scientic publications. Demands related to the interpretation step concern
the challenges to keep track of the data, annotation and analysis steps at all
times and to present the results in a clear and understandable form.
[6.3.4.1] Support for bringing together all data for interpretation[6.3.4.2] Support for linking results to the underlying data source
Demands 6.3.4.1 and 6.3.4.2 derive from the users comments that emphasize the importance of a sound interpretation and its documentation toguarantee reproducibility (cf. Section 4.3.3.3 'interpretation of results' andTable 4.10, Section 5.3.4 and Figure 5.16, and Table 5.10) and point out
related challenges (cf. Section 4.3.6.2 and 4.3.6.3).
6.4

Demands related to best practices

There is a lack of tools support for enforcing a sound methodology incorpus linguistics. First, the demands relate to means for gaining an overall
understanding of the corpus under study by inspecting its design andcompositions, and by evaluating its validity in terms of balancing, representativeness and suitability for a research question. In addition, there is a202need for supporting systematic and targeted checks of the corpus data andannotation quality. Furthermore, there is a need for evaluating the qualityof sets of search results and means to post-process faulty data or to exclude
it from the analysis.
Second, methodological demands are also related to workow management and documentation with the aim to safeguard the reproducibility ofanalyses (cf. Section 6.3.4). This concerns support for the systematic
documentation of the corpus research procedure by recording corpus
characteristics and parameter settings for tools and analyses, and by logging
the dierent analysis steps and related decisions (e.g. classication schemes),recording the dierent analysis cycles (e.g. for exploration or query renement), and by storing intermediate results sets for backtracking.
Last but not least, a framework is needed to support and guide soundand systematic procedures, in particular in response to the skills proles ofcorpus users (i.e., concerning programming and scripting, corpus linguistics
methodology and statistics, etc.). It should assist in the creation of consistent annotation schemes and the calculation of inter-annotator agreement.
Also, it should assist the selection of relevant statistical measures and theestimation of the generalizability of results based on comprehensive information about the coverage and distribution of results, etc.
[6.4.1] Support for monitoring the corpus composition, characteristics, coverage and annotations[6.4.2] Support for evaluating corpus and annotation quality[6.4.3] Support for assessing search results, e.g. errors in data, exhaustiveness, distribution, etc.
[6.4.4] Support for systematic documentation of procedures and tools settings to enable replication[6.4.5] Assistance with applying sound methods and following systematic
procedures
Demands 6.4.1 and 6.4.2 derive from the observation that users aim to understand the characteristics of the corpus they are using (cf. Section 4.3.3.3'data exploration' and 'interpretation', Table 4.4, Section 5.3.4 and 5.3.6.2).
The questionnaire results attested that the task of getting to know the corpus is not among the most applied tasks, but that it is signicantly more
relevant for users with programming skills (cf. Section 5.3.4 and Figure 5.16,
Table 5.13). Demand 6.4.3 relates to the users' need to post-process data tobuild reliable analyses on them (cf. Section 4.3.3.3 'classication of results').
Demand 6.4.4 is derived from the importance of systematic procedures anddocumentation as expressed by corpus users (cf. Section 4.3.6.1, 4.3.6.2 and4.3.6.3, and Table 5.10). In the questionnaire it documentation was named as
the second most important task in corpus procedures, which is signicantly
more relevant for users with a linguistic background than for users with acomputational background (cf. Section 5.3.4 and Figure 5.16, Table 5.13).
Demand 6.4.5 derives from the fact that several best practices for corpus
203

procedures like data annotation, statistical analyses and results evaluation
exist, but not all corpus users are familiar with them (cf. Section 4.3.3.3'classication of results' and Section 4.3.6.3).
6.5

Conclusions

The summary of demands that are presented in this chapter shows that
corpus research practice indeed has a wide range of open subjects thatstill partly hinder the daily work of the corpus users. Not all of the listed
demands are new, and for several of the demands partial solutions can befound. However, it shows that the challenges related to the current corpus
user situation are diversied and solutions can neither be limited to solely
technical or functional aspects, nor are obvious solutions out there forthe more open-ended demands. This indicates that a careful mapping of theopen demands and the current practices and tools is needed to planand dene the next steps for addressing the identied requirements. It also
indicates that more research is needed to come up with novel approaches
and solutions for the less clearly dened demands.
Chapter 7 discusses general implications of the listed demands andpresents rst indications for how to approach their resolution. In orderto do so, it provides directions on how to approach the demands, highlights
a number of relevant ongoing works that are related to some of the demands
and draws sketches of rst innovative solutions for specic tool features that
could respond to some of the demands. This is needed, as the demands are
related to dierent methodological and pragmatic levels of the corpus research practice, and several of the demands formulate open-ended needs
for which it is far from obvious of how they can be addressed.
Chapter 7 does not present an exhaustive analysis or comprehensive design plans for all open demands, but outlines a path and provides
examples for how follow-up works could act on the open demands researched and presented in this thesis.

204

-

Chapter

7-

Potential extensions for corpus tools

205

206

Aim of this chapter

This chapter aims at demonstrating how the open issues and requirements in corpus linguistics as summarized in Chapter 6 could be addressed in the future. It provides rst pointers of discussion for linking open
demands to ongoing trends and developments, and comes up with a preliminary set of ideas and design sketches of potential solutions for corpus
tools in relation to selected demands.
The chapter starts with a general discussion of the results of this userfocussed research and their overall implications for advancements andextensions of corpus tools (Section 7.1). After giving this overall perspective, the chapter discusses needs and potential solutions related to corpus
tools with regards to documentation and training (Section 7.2), technical robustness and usability (Section 7.3), and procedures for data
management and analyses (Section 7.4). Finally, the discussion enters
into the more open-ended functional needs that were brought up by theusers (Section 7.5) and outlines potential paths to their solution. In order toexplore one subject area that could systematically foster the advancementof corpus tools, an additional section (Section 7.6) is dedicated to the use of
linguistic visualizations to support corpus analyses. It describes the basics of information visualization theory and discusses how visualization-based
solutions could help addressing selected demands within corpus tools.
7.1

Innovation potential for corpus tools

Zooming out from the detailed demands in corpus linguistics that aredescribed in Chapter 6, a number of interesting observations regarding theneeds and innovation potential for corpus research tools can be attested. As
described above, users expressed needs related to several aspects of corpus research, ranging from the quality and availability of corpus resources,
over technical and usability needs, up to needs for functional extensions oftools. For the discussion and logical organization of demands, weaimed at being as detailed and exhaustive as possible (Chapter 4 to 6). Thisis reected in the presentation of results of the interviews as well as in thedesign and discussion of the questionnaire survey and its results. However,besides the awareness for dierent needs and their overall importance, bymeans of the interviews we also gained a sense for the perceived weight ofdierent needs in the corpus linguists' daily working context, that is how crucial they are for corpus linguistics to work, which could be further conrmedby the quantitative results of the questionnaire.
The issues that showed to create most concern for corpus users are related
to the availability, quality, and annotation of corpora, as well as to207the successful and ecient operation of corpora and tools in terms ofcompatible data formats and encodings, and in terms of the usability of tools
and query languages.
Indeed, these issues pose fundamental practical challenges, for which
however solutions are mainly known and established, but require large amountsof manpower or specic skills to be implemented. To give an example, therelative scarcity of corpora for lesser-used languages could in many cases besolved, if enough money and/or expert man power would be available. Also,
the usability of query interfaces could be improved by UI designers, if theywould be systematically involved in the development processes. Other examples are borderline, like the need for powerful query languages, since theuse of complex query languages inevitably requires a certain degree of prociency of the user. Thus this need cannot be fully solved on the developers
side only.
Demands related to functional needs for corpus tools are expressedby corpus users to a much lesser extent and in a more abstract or fuzzy
fashion. For example, users indicate a need to 'get an overview of the results'
without being able to specify which kind of tool functionality they would lookfor. Also, users desire to 'compare results across corpora', without specifying
what this comparison should relate to. Finally, corpus users often refer to theimportance of methodological principles that need to be safeguarded, suchas 'working with corpora that are suitable for the given study' and carefully
analyzing that 'results derived from a corpus are generalizable'.
Summing up, this study showed that, on the one hand, corpus linguists
have a number of pressing demands for which mostly solutions are known
on a theoretical level, but are partly still lacking solutions on their practicalimplementation. On the other hand, corpus linguists also have a large setof less prominent demands or visions of what would additionally be
helpful for their work. For the former, this study helped to estimate theweight of the partially unsolved needs. For the latter, this study helped tounderstand what corpus users would like to be able to achieve in addition,
while the specic tool functionalities to support these ambitions still have to
be found, dened and implemented.
Accordingly, we approach the following discussion of potential solutions to open demands in corpus linguistics from dierent angles. In
Sections 7.2 to 7.4 issues are discussed for which solutions are mainly known
and partly worked on, and pointers to examples of related works are provided. In Section 7.5 the more open-ended needs and ideas for innovationin corpus linguistics are looked at, and sketches of tool functionalities toaddress these needs are outlined in an exemplarily fashion. Finally, Section
7.6 dives into the domain of linguistic information visualization anddiscusses how it could potentially benet the advancement of corpus tools.
208

7.2

Documentation, consulting and training

The demand for a centralized documentation of existing corpora (cf. Demand 6.1.1) aims at making existing corpora ndable and enabling accessto meta information (e.g. corpus composition, annotations, data format, licences etc.) which allows for the informed selection of suitable corpora fora corpus study. Providing a centralized access point for corpus resources
further increases the standardization and reuse of corpora. It is one measure for addressing the need for a wide and varied oer of corpora, next tofacilitating the creation of corpora (cf. Section 7.4, Demand 6.1.2), andis in line with the FAIR principles1 which formalize the demand to foster
Findability, Accessibility, Interoperability and Reuse of scientic data.
The increased attention twoards standardization and documentation eorts over the past two decades resulted in the foundation of multiple
large-scale infrastructure initiatives (cf. DARIAH, CLARIN, META-NET,
ELG, elexis, Europeana, etc.) which demonstrate that the creation of centralized documentation of resources has become a shared objective among
empirically-based research domains. Next to individual universities2 , European initiatives like CLARIN are oering comprehensive documentation anddocumentation infrastructure (cf. CLARIN resource families3 and Virtual4
Language Observatory ). Demand 6.1.1 is not fully resolved, but is on a good
way to be met.
The need for assistance with applying sound methods and systematic procedures (cf. Demand 6.4.5) concerns both, the use of automatedprocedures where applicable, and the enforcement of systematic methodological choices regarding best practices in corpus analyses including statisticalmethods, annotation procedures, etc. Support with automated procedures and extended technical guidance could partly be provided by extensive documentation and a user-friendly design of the tools (see Section 7.3).
Also, sophisticated tool features should in the future help guiding the user
regarding processing and analysis choices though attempts in this direction
are still in their beginnings (see Section 7.4). Since decisions regarding theautomation of analyses and their possible repercussions are usually complex,a detailed and interdisciplinary understanding of both the technical aspects
and the linguistic research objectives is often crucial for being able to apply
automated procedures. Therefore, the most promising path to the adoptionof automated procedures are interdisciplinary project teams which allow
for direct consultation among corpus linguists and computationally trained1

https://www.go-fair.org/fair-principles/
See for example https://uclouvain.be/en/research-institutes/ilc/cecl/learner-corporaaround-the-world.html,
https://uclouvain.be/en/research-institutes/ilc/cecl/learnercorpora-around-the-world.html
3
https://www.clarin.eu/resource-families
4
https://vlo.clarin.eu
2209sta, as in fact is often implemented in practice to jointly identify where
automated work is useful and feasible. Since several corpus linguists still experience diculties in obtaining this kind of technical support in a systematic
fashion5 , an online helpdesks as part of infrastructure initiatives like CLARIN(cf. e.g., CLARIN knowledge centers6 ) could open up valid solutions to thisneed . Also, for support with sound methodological procedures both
helpdesks and guided workows within tools (see Section 7.4) could be relevant solutions. They should provide support with corpus design and related
legal issues (e.g. copyright regulations and licences, GDPR and informed
consent, anonymization, etc.), and should assist the workow planning andexecution (e.g., dening annotation schemes, selecting statistical measures,etc.). For the moment, little interactive and hands-on support can be found
for these methodological needs in a systematic fashion.
Related to this, also the oer of diversied training opportunities
should be further strengthened. On the one hand, adapted training in programming/scripting and applied statistics as well as corpus linguistics theoryis already part of university curricula in linguistics or computational linguistics, even though the focus and extent of the courses may greatly varyacross universities. On the other hand, corpus linguistics researchers comewith various professional backgrounds (cf. Section 5.3.1) and therefore
demands for improving skills can extend over the university period. Modular training courses on dierent subjects outside of the university context(e.g. presented as webinars or intense workshops at conferences) could be avaluable option to increase the practical abilities of corpus users. Steps havebeen taken to address demand 6.4.5. in the proposed directions, but may
need to be strengthened to live up to actual demands.
7.3

Technical robustness and usability of tools

Issues with general technical aspects of corpus tools and their usability persist since long and call for reliable solutions. Limitations related to thescalability and portability of tools, and partly also to their interoperability, (cf. Demand 6.2.1) are genuine software development targets, which
can be solved by allocating more resources to development tasks. The issues with interoperability also blend over into standardization eorts (which
exchange formats are expected and desirable) and general workow design
and user needs related to it (see Section 7.4). The general shortage of adequately retributed professional and sustainable development is related to thelimited revenue of specialized and evolving software for research purposes.
Even more than for corpus analysis tools, this is true for NLP tools,
5Interviewees reported on colleagues who occasionally give them a hand with certain
technical needs.
6
https://vonweber.nl/cgi/kcentres_page.cgi

210which do not only face technical issues but also the open research challengeof reliable linguistic processing of various languages (cf. Demand 6.1.3).
However, recent advancements in deep learning approaches show promising
results for modst NLP application contexts. In order to allow for the muchneeded persistence in tool developments, large-scale European infrastructure funding should extend over development works for corpus software and
NLP tool. Also, these developments should, wherever possible, be managed
as open-source community projects to maximise the impact of individual
contributions and adaptations carried out by dedicated research projects
or individuals. As well as for centralized documentation eorts, also here
CLARIN7 is working towards this direction, as well as initiatives like theEuropean Open Science Cloud8 and the European Language Grid9 .
The need for improved usability of corpus tools (cf. Demands 6.2.2)
has to be addressed on a functional level (i.e. search and analysis options,
see Section 7.5) as well as on an interface design level. For the latter, UI
design expertise is needed as well as detailed studies for improving specictool solutions. Similar to software development in general, usability aspects
are often falling short in research software developments for the above mentioned reasons of sparsity of nancial resources. Given the catalyzer eectof user-friendly research tools on the performed research, specic funds for
improving the usability and user experience of software supporting researchtasks are a relevant desideratum, which is currently not met.

7.4

Support for workflow and data management

The demand for support in creating corpora (cf. Demand 6.1.2) tofacilitate the creation of a wide variety of corpora could be addressed byeasy-to-operate pipelines for corpus creation and annotation. Several efforts have been made to automate procedures for corpus creation andlinguistic annotation, including tools for harvesting texts from the web(cf. BootCaT tools10 , SpiderLing (Suchomel and Pomikálek, 2012)), formanaging corpus metadata including parametre settings and versioning information (cf. Laudatio project11 ), and the linguistic processing of texts
with a series of annotation tools (cf. WebLicht12 and its conceptual successor SwitchBoard13 , as well as other language agnostic and language specic7

https://www.clarin.eu
https://www.eosc-portal.eu/
9https://www.european-language-grid.eu/
10
http://bootcat.dipintra.it
11
https://www.laudatio-repository.org/projecthome
12
https://weblicht.sfs.uni-tuebingen.de/weblichtwiki/index.php
13
https://switchboard.clarin.eu/
8211pipelines like UDPipe14 or NLP-PIPE for Latvian15 ). Questions of interoperability (cf. Section 7.3) do apply to corpus creation pipelines and corpus
analyses workows likewise, as they can include several processing steps performed across dierent tools (e.g. search results of concordancing tools areexported and fed into statistical analysis tools). To facilitate transitions
between tools frameworks for glueing together dierent tools and converting input/output formats are needed. Approaches in this direction alreadyexists, like the Apache UIMA framework16 for managing the data ow between independent processing components and frameworks built on top ofit, like DKPro Core (Eckart de Castilho and Gurevych, 2014). More recent
tools to facilitate interoperation are Salt17 and Pepper18 which are both partof the corpus-tools.org toolchain.
With regards to the demand for enforcing a systematic and soundmethodology (Demand 6.4.5) mechanisms for the quality control of the
corpus data under analysis need to become part of corpus processing tools (cf.
Demand 6.4.2, Section 7.5.1). As a baseline, the corpus tool should facilitatechecks on the corpus composition or data quality (cf. Section 7.5.1). Overall, support on technical aspects of the corpus analysis workow should beaccompanied by methodological support, as discussed in Section 7.2 above.
For example, the tool could guide and remind the user to do certain checks
before and while carrying out an analysis (e.g. checking for exhaustivenessof data retrieval, cf. Section 7.5.2). Also, the tool should accommodate
functions to export or save a summary of the current analysis status (e.g.
corpus version, corpus tool, tool settings, search query, set of search results,etc.) at each moment in order to allow for replication as described in thefollowing paragraph.
To address the demand for the systematic documentation of analysis procedures (cf. Demand 6.4.4) infrastructures need to be put into
place that allow to consistently record tools settings and analysis steps
as well as transitions between tools, and data modications applied at transitions (e.g. format conversions). Ideally, the envisioned kind of infrastructure
would also encourage the documentation by the user throughout the analysis workow. By logging and describing searches, search settings and analysis
cycles, as well as parameter settings and how they are adapted along the analysis ow, the persecution of a study, in which all analysis steps and decisions
are fully transparent and replicable would become possible and could evenbe softly enforced. Solutions in this direction have recently been proposed.
Gärtner et al. (2018) present a generic tool assisting the user in recording
process metadata in order to produce a sustainable workow documenta14

http://ufal.m.cuni.cz/udpipe
https://www.clarin.eu/blog/clarin-latvia-introduces-nlp-pipe
16https://uima.apache.org
17
https://corpus-tools.org/salt/index.html
18
https://corpus-tools.org/pepper/index.html
15212

tion in a systematic fashion. Machálek (2020) presents an advanced query
history feature within the Kontext tool. It allows for the longterm storageof queries, including additional information like date and time, queried corpus, subcorpora, query type and parameters. Finally, Brezina et al. (2020)
have recently introduced the Wizard tool to #LancsBox, which allows forthe automatized compilation of PDF research reports. The research
reports are generated based on the queries and analysis options chosen by
the user and include queries, parameter settings and analysis results for acustomizable set of standard corpus analyses (i.e., concordances, frequency
lists, collocation analysis, etc.). For a more sophisticated version of thisdocumentation support the tool could include a number of documentation templates for common analysis steps, like query renement or manual
results classication.
7.5

Functionalities for processing and analysis

In the following subsections, demands related to processing and analysis procedures (cf. Section 6.3) are addressed and suggestions for related
functionalities will be proposed.
When new functionalities are introduced into a corpus tool, dierent
design decisions have to be taken, which we distinguish as follows:
1. the design of the function and what it should do
2. the design of the user interaction to operate the function
3. the design of the visualization of the output
4. the design of the placement of the function within the toolFirst, it needs to be dened fore each function which command will
be carried out by the tool on which data and what results will be returned.
Second, it needs to be dened how the user will operate the command that iswhich user interaction is needed to launch the command. Third, it needs
to be dened how the results of the command will be returned and displayed
to the user that is the visualization of the function's output. Finally,it needs to be dened how the functionality will be integrated with otherfunctionalities of the corpus tool that is the placement of the functionality
within the tool, e.g. if the functionality will be oered as a stand-alone
application, or if it will be presented in the main menu of the tool or withina secondary menu.
In the descriptions in the following subsections the focus is strictly on thefunctional aspect (design aspect 1.). For reasons of clarity, the other three
aspects are presented in the most simplistic way possible. Regarding the user
213

interaction and visualization, we present each function within a standard
GUI-based interface19 and show a most basic results visualization to present
the data20 . For the placement of the functionalities, each functionality ispresented either as part of a main or secondary menu of a most basic webbased corpus search page.
While this basic setting is not the most suitable choice for all functionalities, design considerations and alternative design options are discussed in
the text where relevant.

7.5.1 Evaluating corpus quality and suitability
For any meaningful corpus study it is relevant to evaluate the validity andsuitability of a corpus for the envisioned analysis (cf. Demand 6.4.1) as
well as the quality of the corpus and its annotations (cf. Demand
6.4.2). In order to support the user in gaining a detailed understanding of a
corpus in these respects, we propose tool features which provide descriptive
documentation, representative samples of the corpus data, and statistics fordierent linguistic categories of interest in relation to the corpus as a wholeor specied subparts of it.
We claim that dierent tool functionalities should allow to retrieve thefollowing information on demand: Composition of the corpus in terms of metadata categories and theirquantitative representation within the corpus (Figure 7.1 provides ashort example for this feature).

 Documentation of the corpus processing and versioning, including which

tools with which parameter settings were used for which annotations,manual interventions, etc.

 Text samples for any subset of the corpus based on metadata lters(including the most representative as well as outlier texts, e.g. in termsof text length, complexity, etc., Figure 7.2 provides a short example
for this feature).

 Annotation categories, their values and quantitative distribution as

well as annotation gaps.

 Comparison of multiple annotations, if applicable (e.g. inter annotator

agreement, agreement/divergence between dierent annotation tools,
automatic annotations and manual gold standard, etc.).

 Heuristics for checking text and annotation quality, e.g. out-of-vocabulary

words, words with varying annotations (e.g. dierent part-of-speechtags), random samples of instances for each tagged unit, etc.

1920

As known from common search webpages.
As known from chart diagrams in standard oce software.

214

All the information should be comparable for dierent corpora or subcorpora,and if available also in relation to a reference corpus.
In addition to evaluating the quality and characteristics of a corpus asa whole, users are also in the need to assess the local data quality andcharacteristics of search results (cf. Demand 6.4.3). Accordingly, tools
should provide support for evaluating dierent aspects of search results, e.g.
based on automatically generated representative samples: The text quality of search results and their annotations.
 The precision and recall values of search results.
 The distribution of search results over the corpus as a whole as indi-cator of their representativeness.

 The t of search results for specic statistical tests (e.g. in relation to

the sample size or distributional characteristics of search results, etc.).

7.5.2 Improving search and export
Users call for powerful, exible and user-friendly search, display andexport functionalities (cf. Demands 6.2.2 and 6.3.1.1), which should accommodate corpora of dierent scripts and with possibly multi-layer andmultiple annotations. Much research has been invested into the representation, search and display of annotated corpora (cf. e.g. ANNIS21
and MMAX222 ). In addition to the conceptual and technical challenges
of search and results representation for corpora with complex annotations,also the user interactions needed for operating related tools (e.g., by using
complex query languages) pose a considerable challenge from the usability
perspective.
The contradicting need of exible/powerful (Demand 6.3.1.1) and easyto-use (Demand 6.2.2) query interfaces can only be approached by combining predened search menus and command lines for entering complex
query denitions or even small snippets of programming code. In fact query
tools should possibly be integrated with programming API's to extend theirsearch capacities. In the same way, also for the presentation of results
and their dierent annotation layers, a combined model is desirable. Userfriendly menus could allow to select or deselect dierent annotation layers
for display and choose among dierent modalities like color, highlighting
by typeface etc. In addition, tools could allow for applying user-dened
schemes for rendering the data exibly in relation to the users' specic preferences. With the ANNIS tool relevant steps towards the provision of dierent and customizable visualizations have been taken. It uses pluggable JAVA
2122

https://corpus-tools.org/annis/
http://mmax2.net/index.html

215

Function: Ccomposition-inspect
The function Ccomposition-inspect is part of the main search menu.
It allows the user to inspect the composition of a chosen corpus in relation toa selected metadata category. Both the corpus and the metadata categoryof interest, such as the temporal dimension (i.e. year or decade), the text
type (see this example), the authors of the texts or characteristics of them(e.g. pupils, university students, professional writers, etc.), can be selected
via simple drop-down menus.
A click on 'get corpus statistics' opens a new window with a diagram showing
the corpus composition measured in number of texts, number of tokens andaverage text length.

Figure 7.1: Example: Search and display of the corpus composition

216

Function: Csamples-inspect
The function Csamples-inspect is part of the main search menu. It allows the user to retrieve characteristic samples of source text from a selected
corpus for inspection. The process is started by selecting a corpus from adrop-down menu and by indicating any combination of metadata lters theuser would like to apply. Metadata lters are entered into the text box in thefollowing form: author='Carl Loris', year='1974-1997', type='novel'. After,
the user can chose the number of texts per sample and the criteria to consider
for sampling (e.g. text length, text complexity, vocabulary variety, etc.) bychecking the provided check boxes. The default values are 5 texts sampled
by text length.
A click on 'get sample texts' opens a new window with three folders containing some of the most simple, most complex and average medium texts out of
the selected and possibly pre-ltered (by metadata) corpus. Hovering overa folder shows a preview of the corpus texts in minimized mode.

Figure 7.2: Example: Search and display of sample texts
217

modules to visualize data23 and also provides a HTML visualizer module24
which allows to create tailor-made visualizations. Similarily, the CorpusExplorer25 (Rüdiger, 2018) allows to integrate customized visualizations basedon HTML5/JavaScript/CSS by means of its 'HTML5-Labor' and gives examples for visualizations created with HTML5 and D3.js26 . Support forexibly adjusting display formats is relevant both for both the inspection of
the data for analysis purposes (cf. Section 7.5.3) as well as for the export ofsearch results (Demand 6.3.1.3)
Another challenge with corpus searches is the denition of searchheuristics (cf. Demand 6.3.1.2), which result in high precision and recall
values for the search targets. Depending on the nature of the search target,
the quality of the corpus and its available annotations, search heuristics canbe more or less dicult to dene and more or less obscure to evaluate. In
order to support the exhaustive retrieval of a given search target and to raiseawareness about precision and recall values of results, we propose to enhancesearch tools by exploration and monitoring functions, which should allow to: Record search denitions and results for each search attempt (Figure

7.3 provides an example).

 Automatically compare result sets for dierent search cycles (see Figure

7.3 for a basic example) and in relation to a gold standard (see Section
7.5.3.2).

 Automatically generalize or rene search queries (e.g. move from spec-ied word form to part-of-speech or vice versa).

 Manually exclude bad results and learn from the manual classicationeorts (e.g. by deriving patterns or using them as training data).

 Calculate precision and recall values on results based on a gold stan-

dard (see below).

 Restrict searches to representative sub-samples of a corpus.

The tool should assist the creation of gold standard data by generating
representative samples of the search results and corpus base data for manual evaluation and processing by the user. The tool should also allow forstoring the manual annotations as project-specic data (i.e. temporary fora specic study) or as additional annotations of the corpus (i.e. re-integrate
annotations into the corpus, see Section 7.5.5 for further discussion).
23

https://corpus-tools.org/annis/visualizations.html
https://corpus-tools.org/annis/resources/ANNIS_HTML_Vis_Guide.pdf
25
https://corpusexplorer.de26
https://d3js.org/
24

218

Function: Cquery-compare
The function Cquery-compare is part of the main search menu. It operates
on a history of recorded queries. By default queries are not recorded but theuser can record queries by clicking the checkbox 'record this search' before
starting a query. For all recorded queries the user can choose to repeat the
search ('re-query'), to eliminate a record ('delete') or to perform a pairwise
comparison of subsequent searches ('compare search results pairwise').
A click on 'compare search results pairwise' returns two tables. The rsttable gives an overview of the queries, their number of hits and links to theirconcordances. The second table provides comparative data for each pair ofsubsequent queries. It shows the number of joint hits (overlap), and thenumber of disjunct hits for each query (q1 without q2, and q2 without q1),
and provides links to the related concordances. In addition, it lists the mostfrequent matches in all three result sets.

219

Figure 7.3: Example: Record and compare dierent search cycles

7.5.3 Exploration of corpus data
7.5.3.1 Exploration of global corpus data
Support for corpus exploration through data-driven and open-ended
analyses (cf. Demand 6.3.2.1) should be provided by oering alternative
ways of corpus access that are not query-based, but can help to identify orrene search targets. Non query-based entrance points to corpus analyses
could be built on a panorama of detailed statistics and descriptive summaries
of various features of a corpus as a whole, a subcorpus or a comparison ofa (sub)corpus with a reference corpus. Relevant features for such kind of
corpus panorama could include but should not be limited to: Spectrum and occurrence variability of dierent corpus features (Figure

7.4 and Figure 7.5 provide a simple functionality for querying corpus
features and the related display of results), such as

1. (a) annotation categories, like part-of-speech,
2. (b) text complexity features, like text length, phrase length, average word length,
3. (c) text types, as dened by metadata, e.g. author, year of publication, genre, etc.,
4. (d) n-grams and collocations,
5. (e) vocabulary, etc.
 Position, distribution and dispersion of words, word classes, n-grams,

collocations, etc.

 Aggregation of contextual patterns around higher level units and con-structions, e.g. named entities or noun phrases.

7.5.3.2 Exploration of focussed search results
Another need for exploration support exists on the level of the explorationof search results, which is relevant to compare and detect patterns and to
identify outliers in the data (cf. Demands 6.3.2.2 and 6.3.3.1), i.e. to reneresearch hypotheses or to identify relevant examples.
To facilitate pattern detection, corpus tools should allow to group andcreate aggregated views of the results based on metadata characteristics,
context structure on dierent levels of description (word/token and part-ofspeech sequences, syntactic structure, etc.) or frequency and distributional
characteristics of the context vocabulary as well as hand-picked features.
With WordSketches27 , the SketchEngine tool is oering one type of aggregated data view calculated on data from entire corpora.
27
https://www.sketchengine.eu/guide/word-sketch-collocations-and-wordcombinations/

220

Function: Cfeatures-inspect
The function Cfeatures-inspect is oered in a secondary menu of the
corpus tool. It allows the user to explore the corpus from a structural perspective by retrieving statistics and examples of selected corpus features.
The Cfeatures-inspect menu allows the user to chose the corpus froma drop-down menu and to indicate all features he is interested in by selecting the respective check boxes. The features include annotation categories,
metadata categories as well as text measures such as text length, text complexity, etc. and frequency distributions of words, lemmas and n-grams.
A click on 'generate feature spectrum' returns a table (see Figure 7.5) displaying the feature spectrum with (1) statistics on the entire corpus (i.e.
number of types, tokens and texts), (2) absolute and relative token frequencies for all annotation and metadata categories, plus text frequencies for
metadata categories, and (3) minimum, maximum and average value for all
text measures and words lemmas and n-grams, plus their averages divided
by quartiles.

Figure 7.4: Example: Overview of dierent corpus features

221

Results display of spectrum of corpus features:

Figure 7.5: Example: Overview of dierent corpus features - results
In addition, tools should provide comparative analyses of the aforementioned aggregated data for dierent but related result sets, e.g. search
results across dierent corpora, search results of complementary linguistics
constructions, search results with dierent context constraints28 or simply
search results that were classied into several groups by the user. Figure 7.6
provides an example of a functionaly for comparative analyses of one corpus
query across two corpora.

7.5.4 Interpretation and reporting of data
Tools should support the close inspection of single search hits (Demand
6.3.2.3) and should generally facilitate data interpretation by bringing together all data for interpretation (Demands 6.3.4.1) and by linking
corpus examples to their underlying sources (Demand 6.3.4.2). Meansto look into the data in detail helps identifying representative and meaningful examples to illustrate analyses, and to support data interpretation incontext. The support can be based on combining dierent views on thedata, for example wider context, frequency characteristics, aggregated information and links to the source text, as has been proposed by Culy and Lyding (2011) with Corpus Clouds. Also, advanced sorting options, based onuser-dened information as well as context and descriptive statistics would
be helpful to highlight the most relevant and representative examples. Possibly, the information extracted from the corpus could be integrated with
corpus-external resources, such as frequency lexica, concept networks,
28

E.g. search term occurring in sentences with more or less than ve words

222

Function: Qresults-compare
The function Qresults-compare is oered in a secondary menu of the
corpus tool. It allows the user to compare results of one same corpus queryin two dierent corpora. The user has to choose two corpora via drop-downmenus and select the features to compare for the two result sets via checkboxes. The features relate to patterns in the composition of the query andits context as well as the distribution of hits across dierent corpus texts anddierent metadata categories.
A click on 'start comparing searches' generates tables that display the mostfrequent patterns of the selected features and compares their occurrence
frequencies across the two corpora.

Figure 7.6: Example: Comparative analyses of searches for exploration223

etc. Also, the exibility in display and export options (cf. Demand
6.3.1.3, Section 7.5.2) can greatly facilitate the interpretation and reportingstep.

7.5.5 Manual data correction and processing
Corpus tools should allow users to dynamically apply manual post-corrections of corpus texts and annotations as needed (Demand 6.3.2.4). Users
should be able to edit erroneous text and annotations within corpus search
results and upload the amendments into the corpus on click. This would
allow the user to improve the corpus while using it, and especially tocorrect errors on-the-y, when encountering them within the results of the
corpus query under study. As soon as a user would submit a correction, thecorpus tool should take care of the technicalities for updating the corpus in
the background (e.g. encoding, re-indexing, versioning, etc.).
Corpus tools should also assist the user in the management of search results, such as allowing for the automatic and manual sorting and grouping of search hits, for example by draging-and-dropping them into dierent
baskets. This can serve to classify results or to reduce result sets (cf. Demand 6.3.3.2). Related to this demand the tool should support the creation
and adjustment of classication schemes, both from the technical perspective (e.g. accommodating multiple or overlapping annotations, adding
labels to specic parts of the search result) and from the documentation
perspective (e.g. dening classication guidelines, documenting successive
modications to classication schemes) (cf. Section 7.4).

224

7.6

Linguistic visualizations for corpus tools

The eld of Linguistic Visualization ('LingVis'29 , (Butt et al., 2020)) has
gained increased attention over the past decade.30 It researches and presents
works that apply dierent types of visualizations to language data in orderto facilitate its analysis or to help better communicate information
about it. Visualizations are also employed as interactive means to accessand modify linguistic data.
Analysis, inspection and manipulation of data, all of these application
contexts are highly relevant for corpus linguistic research. Linguistic visualizations oer new representation formats that can accommodate potentially complex data of large quantities in information-dense displays. Therefore, they bear the potential to add a new dimension to the analytical functionalities of corpus tools. In fact, visual representations have increasingly been introduced into corpus tools in the past years (see Section 7.6.3)
and the questionnaire investigation on visualizations showed that users arevery positive about the potential use of visualizations in corpus tools(see Section 5.3.6). However, most of the existing visualizations found in
corpus tools use well established graphical representations and are not particularly targeted to the visualization of language data. Exploring further
the possibilities of linguistic information visualization for corpus linguistics
are expected to open up further opportunities of corpus research support. In
particular, several of the open-ended demands related to data exploration
and understanding as discussed in Section 7.5 could be addressed by visualsolutions.
In the following subsections we will introduce the theory and state-ofthe-art of linguistic information visualization, and will present and discuss
practical examples for application in corpus tools. In Section 7.6.1, we will
provide an overview of the basic assumptions and principles of information
visualization theory. In Subsection 7.6.2, we will give a short overview
on the state-of-the-art in linguistic visualization research. In Section
7.6.3, we will discuss its potential for corpus linguistics and summarize
to what extent it has been adopted within corpus tools up to date. Finally,in Section 7.6.4, we will discuss potential visualization solutions for twoof the open demands discussed in Section 7.5 above.
29

Also
termed'VisLing'(http://ilg.usc.es/VisLing/),'LInfoVis'(www.eurac.edu/linfovis), 'LangVis', etc.
30
Demonstrated by an increasing number of publications and workshops, i.e. 2009:ESSLII workshop 'Linguistic Information Visualization', 2012: EACL workshop 'Visualization of Linguistic Patterns', 2014/2016/2018: LREC workshops 'VisLR: Visualization
as added value in the development, use and evaluation of LRs', 2015: Nodalida workshop'QueryVis - Workshop on Innovative Corpus Query and Visualization Tools', 2016-2021:#VIS4DH 'Workshop on Visualization for the Digital Humanities', etc.

225

Figure 7.7: Visual variables after Bertin (1983), as shown in Carpendale(2003), Table 1

7.6.1 Information visualization theory
The eld of Information Visualization (IV) is dened as '[t]he use ofcomputer-supported, interactive, visual representations of abstract data toamplify cognition' (Card et al., 1999). It distinguishes from other types of vi-

sualizations, e.g. scientic visualization and geovisualization, by focussing on
abstract data, which has no natural or inherent physical correlate. Abstract
data types include, among others, statistical and distributional information,
process descriptions, as well as language data.
Information visualization research is concerned with principles and techniques for the creation of meaningful graphics from data. These principles
and techniques build upon the special visual capabilities of the human's perceptual system (Ware, 2004; Hearst, 2009), as researched in cognitive scienceand psychology. In fact the human perceptual system is particularly strong
for processing visual input, and can quickly perceive visual dierences inform, color, etc. (see below). The common English proverb 'A picture is
worth a thousand words' expresses this insight in simple words.
In a foundational work, Bertin (1983) has proposed a set of visual variables (see Figure 7.7) for distinguishing visual properties of graphical representations.
Among the visual variables position indicates the placement of thevisual element within the twodimensional (or multidimensional) space. Size226

indicates the spatial measures of the visual element. Shape indicates theform of the visual element. Value indicates the brightness of the color of thevisual element. Color indicates the hue of the visual element. Orientation
indicates the relative directivity of the visual element. Texture indicates theinherent visual structure of the element.
Bertin (1983) further denes dierent characteristics of the visual variables, which allow to distinguish their concrete instances. Selectivity andassociativity dene whether variables are dierent or similar among eachother. Order denes the ranking between variables, e.g. in terms of quantity, quality or intensity. Quantity allows to indicate to what extent variables
are diernt from each other. Finally, length denes the maximum numberof meaningful distinctions within each variable. These numbers are related
to the humans' perceptual capacities for processing and are drawn from perception research. Also, while the human eye is able to make ne grained
distinctions of, for example, millions of colors, in visualization design thenumber of dierent colors that are recommended to be used for distinguishing elements is limited to even less than 10 dierent colors which can be
distinguished at a glance.
Information visualization guides the process of translating information into visual form to create eective visualization applications. Card et al.
(1999), based on Chi (1999), has presented a reference model for visualization (see Figure 7.8) that formalizes the transformation from raw data
to views, by distinguishing a process of three steps: (1) data transformations from raw data into structured data tables, (2) visual mappings fromdata tables to visual structures, and (3) view transformations from visual
structures into modiable views.
Data tables organize the data and explicitly encode relevant characteristics, such as e.g. occurrence frequencies of certain words in a corpus. Visual
structures are explicit specications of graphical representations. They dene which graphical elements to use and the selection of their visual properties. Views dene the nalized graphics for display by setting the concretevalues for graphical parameters such as the rendering of color, scaling andposition.
Eective visual mappings dene which visual structures can be used
to create visualizations that suit the specic characteristics of the data andits informational intent in the best possible way. The visual mappings are
informed by the set of available visual properties of graphical elements andon visualization principles of how to organize the elements. Due to thevariety of data types and informational purposes, it is not possible to denegeneric mapping rules or even an exhaustive typology of visual mappings.
Any visualization design is context-dependent, but should be oriented bygeneral visualization principles. In the following paragraphs a selection ofbasic visualization principles is given (based on information compiled andpresented in Lyding and Culy (2012)).
227

Figure 7.8: Reference model after Chi, as in Card et al. (1999, g. 26.10)
Expressiveness and purposefulness. In order to be meaningful, visualizations need to encode all the relevant and only the relevant information(Tufte, 1990). All graphical elements of a visualization are expected to carrya specic and purposeful meaning. As expressed by Kosara (2009) 'Visualization has to be more than pretty pictures.

It has to inform.

challenge. It has to further our understanding.'.

It has toThis purposefulness also
regards transmitting the informational intent of the visualization, e.g. byhighlighting the most important information.
Coherence and consistency. As described by the 'Gestalt principles' in psychology (Todorovic, 2008) the visual conguration of elements
is decisive for how groupings of elements are perceived. For example, similar elements or elements placed close together are understood as forminga group. Or as Tufte (2006) has phrased it: 'Sameness of a visual element
implies sameness of what the visual element represents'. Therefore, meaningful visualizations need to be consistent with regard to relations of similarity,
proximity and conguration, and avoid 'eects without causes' (Tufte, 2006).
Conventions. Visualizations need to respect established conventions
concerning their use. For example in cartography, blue color is used toindicate water, or in trac signs red color indicates a warning or prohibition.
Conventions can be culturally dependent and have to be used accordingly.
Clarity. Minimal and dense visualizations allow to reduce the cognitiveeort, while 'clutter and confusion are failures of design, not attributes of
information' (Tufte, 1990). Accordingly, Tufte (1990) suggests to 'let thesame ink serve more than one informational purpose'.
Transparency and integrity. Visualizations should provide aids fordecoding their informational content through explicit indications (e.g. labels
and legends) and contextualization (e.g. using scales, rulers). Furthermore,228

it needs to be made explicit what information got included or left out inthe visualization. This also concerns the traceability of transformations andabstractions of the underlying data (Tufte, 1990).
Perspective and arrangement. Visualizations need to provide dierent perspectives on the data as has been expressed by Shneiderman (1996)'s
visualization mantra: 'Overview rst, zoom and lter, then details on demand'. Within a single visualization, various aspects of the data have to bevisually stratied by layering and separation (Tufte, 1990) and the display
space should give most room to the data aspects of the highest interest (Cardet al., 1999).
Interactivity. Implicitly included within Shneiderman's Mantra, visualizations should make ecient use of interactive features, as 'rapid interaction fundamentally changes the process of understanding data' (Card et al.,1999). Interaction techniques include panning, brushing and linking, detailson-demand, animated transitions, etc.

7.6.2 Linguistic visualization examples
Over the past decade, a multitude of linguistic visualizations have beendeveloped and presented at scientic venues and workshops at the crossroadof information visualization, information technology, NLP, language technology and linguistics.
Depending on their informational intent, related to content (i.e. information retrieval31 ) or linguistic structure (i.e. linguistic analysis), thevisualizations build on dierent aspects of language data, including: Documents and keywords in documents,
 Linguistic categories (e.g. parts-of-speech for words, syntactic consti-

tutents within sentences, phonemes and intonation markers, etc.) andlinguistic relations (e.g. syntactic dependencies, collocations, etc.),

 Sequences (n-grams, MWEs, phrasal constructions, etc.),
 Occurrence frequencies (of words, constructions, etc.).

Document collections and keyword analysis.
A good share of language visualizations is made up of visualizations for document and keyword analyses or topic modelling. Related works use
spatial displays like galaxy and stareld views that show the distribution
and semantic proximity of documents and terms. Examples are listed in
31

In relation to visual text analytics, Kucher and Kerren (2015) have presented a comprehensive survey of text visualization techniques based on more than 400 related research
papers. The survey results are presented as taxonomy of text visualizations and are browsable on an interactive website (https://textvis.lnu.se/).

229

Figure 7.9: Display of the IN-SPIRE document explorer tool, as of https:// in-spire. pnnl. gov/ index. stm

Annex C and include GistIcons (DeCamp et al., 2015) see C.1, FacetAtlas(Cao et al., 2010) see C.2, TextArc (Paley, 2002) see C.3, InfoSky (Granitzer
et al., 2004) see C.4, xHIPP (Dias and Minghim, 2018) see C.5.
Building on these visualizations a number of workbenches with multiple displays have been proposed, which also include features like date,genre of the texts or occurrence frequencies and collocation strengths ofterms. IN-SPIRETM 32 (see Figure 7.9), Jigsaw (Görg et al., 2013) see C.6,
MiTextExplorer (O'Connor, 2014) see C.7 and FeatureLens (Don et al., 2007)
see C.8 are examples of such interactive visual analysis systems.
Docuburst (Collins et al., 2009a) (see Figure 7.10) visualizes document
keywords in relation to semantic taxonomies and the TextWheel system by
Cui et al. (2012) visualizes macro and micro relations between keywords
and articles in news collections (see Figure 7.11).
ThemeRiver by Havre et al. (2000) (see Annex C, C.9) is an early famous
example that visualizes thematic variations within document collections
over time and Parallel Tag Clouds by Collins et al. (2009b) (see C.10) oers
a comparative visualization of the most frequent tags in document
collections over time.

32

Pacic Northwest National Laboratory, https://in-spire.pnnl.gov/aboutus.stm

230

Figure 7.10: Docuburst visualization, as dened in (Collins et al., 2009a),
Fig. 7, p. 1045

Figure 7.11: TextWheel system, as presented by Cui et al. (2012), Fig. 1,
p. 2

231

Figure 7.12: Display of the Convis tool, as dened in (Hoque et al., 2014),
Fig. 1, p. 48

Conversation and discourse analysis
Another set of visualizations focuses on the linear structure of texts orinteractions between agents in conversations. For example Zhao et al.
(2012) (see Annex C, C.11) propose the DAViewer for analyzing discourse,
while Viégas et al. (2006) (see C.12) have proposed the Themail visualization for analyzing relationships in email conversations and Wilhelm
et al. (2013) (see C.13) for analyzing speaker contributions in Shakespeare plays. Hoque et al. (2014) present the Convis visual analytic tool(see Figure 7.12) that provides support for multifaceted exploration of asynchronous conversations, as found in online conversation threads. Angus
et al. (2012) (see C.14) present a recurrence plots as an information visualization technique to reveal recurrent concepts in conversations analyzed
as complex time series data. Kiesel et al. (2020) present a visualization foranalyzing argumentation ows in essays (see Figure 7.13).

Linguistic features
A rather smaller number of works is concerned with the visualization of particular linguistic features. For example, Mayer et al. (2010) have proposeda visualization of vowel harmonies and consonant co-occurrences (seeFigure 7.14). Herold and Stathi (2007) (see Annex C, C.15) have used visualization techniques to analyze the syntagmatic xedness of multi word
expressions. Other visualizations focussing on syntactic structures include DependencyDiagrams (Culy et al., 2011) (see C.16) and visualizations
232

Figure 7.13: Visualization of argumentation structure, as dened in (Kiesel
et al., 2020), Fig. 3, p. 3

Figure 7.14:

Display of Vowel harmony, as presented in Mayer et al.

(2010), Figure 1, p. 16

of syntactic tree structures (Meurer et al., 2016) (see Figure 7.15).

N-gram and concordance visualizations
Also for running text a growing number of visualizations have been proposed. Word Tree (Wattenberg and Viégas, 2008) is an innovative example of an interactive and information-dense visualization of concordancelines (see Figure 7.16). DoubleTree (Culy and Lyding, 2010) (see C.17)
and Netspeak Wordgraph (Riehmann et al., 2011) (see C.18) have extended
the idea of Word Tree by providing context on two sides and highlighting recurrent word sequences. Sheehan (2014) proposes a Mosaic viewon concordances and Cheng et al. (2006) focus closer on the analysis ofrepetitive patterns in n-grams and n-grams with gaps ('skipgrams').
233

Figure 7.15: Treebank visualization of c- and f-structures, as presented in
Meurer et al. (2016), Figure 4, p. 4234

Figure 7.16: Wordtree visualization, as presented in Wattenberg and Viégas(2008), Figure 10, p. 1227

Slash/A (Todorova and Chinkina, 2014) and interHist (Lyding et al., 2014)(see Figure 7.17) present visualizations of n-gram patterns. van Ham et al.
(2009a) present Phrase Net to visualize collocation relations (see Figure7.18).

Lexical visualizations
Finally, several visualizations have been developed that focus on lexicalunits and their occurrence frequencies. The Glance system (Chen et al.,2014) visualizes lexical phenomena for language learning by combiningdierent modules for creating standard visualizations like bar and pie chartsand sunburst diagrams. Anderson et al. (2002) show a visualization forquery term frequencies based on pie charts. In addition word and tag
cloud visualization are omnipresent representations of lexical units andtheir occurrence frequencies, which are found on may websites and in evenin some corpus tools (cf. Section 7.6.3), despite the fact that they have been
criticized for fundamental shortcomings of their visual design.33

To conclude, up to now most visualization eorts related to language data
make use of existing information visualization techniques (e.g. tag
clouds, starelds, galaxies, charts, matrix views, etc.). On the one hand, thereuse of existing information visualization techniques is sensible for one partof the language data, namely numerical or nominal derivatives of language data, such as frequency information of language items, categoricalinformation (e.g. labels for grammar and semantic categories) or relational
data (e.g. links among collocations or otherwise related words). On theother hand, new visualization techniques, which are specically designed
to present genuine characteristics of language data, which cannot be
abstracted to numerical or categorical features easily. For example, the lin33

cf. e.g., https://towardsdatascience.com/word-clouds-are-lame-263d9cbc49b7

235

Figure 7.17: interHist visualization, as presented in Lyding et al. (2014),
Figure 2, p. 638

Figure 7.18:

Phrase Net visualization, as presented in van Ham et al.

(2009b), Figure 2, p. 4236

earity of word sequences, the coherence of text ows, and the semantics ofposition information within text sequences are types of information that are

inherent to language as informational code. Also, the presentation ofgrammar paradigms and related exceptions as well as the idiocracy of themeaning of words or text excerpts pose particular challenges for visualization. Finally, language data inhibits a complex interplay of structuraland frequency information that requires analyses on dierent levels of
abstraction to be observable. For example, the grammaticality of phrasal
constructions can be judged based on a single occurrence of a phrase, while
the degree of syntagmatic xedness of a phrase requires both the occurrenceof its structural form as well as information on its frequency of occurrence.
Only a smaller fraction of the linguistic visualization techniques discussed in this section are particularly designed for non-numerical andnon-categorical characteristics of language data. They all relate to thelinearity of text and recurrent patterns within the linear sequences, and were
described in the paragraph about 'n-gram and concordance visualizations' above.
Thus, while the interest for adapted visualizations for language
data has grown over the last ten to fteen years, many more languagespecic visualizations could be envisioned in the future. In particular, anintensied focus on structural characteristics of language data as
opposed to content-related aspects would be desirable to complete the pictureof meaningful visualizations for language data.

237

7.6.3 Visualizations in corpus tools
To a limited extent, visualization features are found in many state-ofthe-art corpus tools. The visualizations range from features like color coding of words and bar charts displaying frequencies, over network graphs fordisplaying collocations and word relations to a few customized visualizations. Table 7.1 provides a brief summary of the visualizations oeredin some of the most widely used (and maintained) corpus tools nowadays,
namely ANNIS (Krause and Zeldes, 2016), AntConc (Anthony, 2020), CQPweb (Hardie, 2012), english-corpora.org34 , #LancsBox (Brezina et al., 2020)and SketchEngine (Kilgarri et al., 2014).
Compared to the vast selection of complex linguistic visualizations
that have been proposed over the past two decades (cf. Section 7.6.2), thevisualizations found in corpus tools are prevalently limited to basic andestablished visualization types (e.g. bar charts and dispersion plots).
Also on the functional level, visualizations in corpus tools only play amarginal role in supporting analyses. They mostly oer a static visual
alternative to frequency tables, with no or limited interactive features or
options for dynamically adjusting parameters to alter the visualization.
In addition to the well-known tools listed in Table 7.1, with the CorpusExplorer35 Rüdiger (2018) is providing a corpus tool that includes somemore advanced and adaptable visualizations as well as functional provisions
to the functionalities by custom visualizations (cf. Section 7.5.2). Still therange of visualizations currently oered by the tool remain limited thoughmore extensive than in other tools. Next to dierent uses of color coding fordata comparison, and line charts for displaying frequency distributions, theCorpusExplorer includes somes examples of mind map visualizations, word
clouds and heat maps.

34
35

www.english-corpora.org
https://corpusexplorer.de238

Table 7.1: Visualizations in the most popular corpus tools
ANNIS

AntConc
239

CQPwebenglish-corpora.org#LancsBox
SketchEngine

color coding
POS in KWIC, colored lines indicating
constituentsword position in
KWIC
colored word clouds
word classes and frequency ranges

POS in KWIC, comparison of collocations

chartsbar chart indicating
frequency of hits,
grid viewgraphsplots
dependency and constituent tree view,
rethorical structure(RST), word alignment by lines
dispersion plot ofhits in texts
bar chart for hits fredispersion plot of
quencies over corpus
hits (beta), lexical
categories
growth curve (beta)
bar charts for hits
distribution
bubble chart for hits collocation graphs boxplot for hits frefrequencies over cor- and networks
quency distribution
pus lesover corpus lesbar chart of hits dis- bubble chart for dif- icons to indicate
tribution, frequency ferent word relations, trends, visualizationbars
thesaurus
for comparing word
sketches7.6.4 Future work for visualizations in corpus tools
The overview of the visualizations found in corpus tools (Section 7.6.3)
showed that corpus tools mainly oer well-established visualizations36
for the representation of frequencies, distribution and relational information,
apart from a few exceptions which experiment with techniques known frominformation visualization research, such as (dynamic) network graphs,
heatmap displays or word clouds. In addition, initiatives like TAPoR, the37 and the Text Visualization Browser 38
Text Analysis Portal for Research
have focused on bringing together visualization techniques and visualization
tools for text analysis, which partly overlap with requirements of the corpus usage scenario but are mostly oriented towards content-oriented text
analysis.
The close match of linguistic visualization purposes and the corpus
linguistics research approach (cf. Section 7.6 'introduction' and 7.6.1),
suggests that corpus tools could greatly benet from intensied eorts tointegrate existing research on linguistic visualizations with corpus tool functionalities in order to extend the functional power of corpus tools. Indeed,
the wide variety of available visualization techniques and linguistic visualizations examples nowadays (cf. Section 7.6.1 and 7.6.2), provides
room for corpus tool developers to explore more and dierent visualizations
to support corpus analyses. In particular, the potential of visualizations to
support data inspection and data exploration tasks, though considered
generally useful by corpus users (cf. questionnaire results in Section 5.3.6),
has not been extensively exploited yet.
In order to advance powerful visualizations to address demands in
corpus analysis procedures (cf. Section 6.3), three challenges need to be
tackled.
First, the corpus linguistics community needs to clearly dene andexpress their research needs with regards to corpus analysis tools. By investigating which data characteristics are relevant for corpus analyses andwhich informational aims drive the corpus analysis process, this thesis provides a starting point for developing a rened set of demands for visualization functionalities, which needs to be extended over time.
Second, and building on the rst point, existing linguistic visualizations (cf. Section 7.6.2) need to be visited by corpus experts in order toevaluate which of those language visualizations could be relevant for corpus
linguistics tasks, and how they could be adapted and included into corpus tools. Indeed, on the one hand not all of the visualizations that havebeen proposed for language data are relevant for the informational aims in
corpus linguistics. On the other hand, there are several visualization tech36

such as line charts, color highlighting and dispersion plots
http://tapor.ca
38
https://textvis.lnu.se/37240

niques that have not made their way into corpus tools, even though theymight be adapted to the characteristics of corpus data and to the analyticalaims of corpus linguistic research questions.
Third, visualization designers need to get involved to create new linguistic visualizations that are particularly adapted for bringing out specic
aspects of the data and thus can be included into corpus tools for supporting
related corpus analysis tasks.
Overall the eorts to advance powerful visualizations should lead tothe integration of more, more sophisticated and more interactive visualizations into corpus tools. In particular, new visualization functionalities
should be closely integrated with corpus search and analysis functionalities
instead of being just another output data format. By being more thanstatic displays visualization functionalities could support the interactive
exploration and analysis of corpus data, by allowing the user to takedierent views on the data or to navigate between parallel displays, continuously modifying both the target of the analysis (by querying and ltering) as
well as its visual presentation form. Thus, innovative visualizations for corpus tools should become analysis functionalities in their own right otherthan just alternative graphical outputs of existing functionalities.

7.6.5 Two examples
In this section we will discuss how two open corpus analysis demands
could be addressed by means of interactive visualizations and what would
need to be considered for choosing the most apt visualization designs for
related data types and informational needs.
By running through two examples, we want to describe the process ofchoosing a visualization for a dened corpus analysis task and of planning how it could be implemented within a corpus analysis functionality.
It will give a rst exemplary outlook on the aims and potential of integrating
advanced visualizations into corpus tools. The examples will present an illustrative sketch on potential visualization designs. Detailed user studies
on the proposed visualizations would be needed to identify the most relevant
solutions and evaluate their practical use for corpus analyses.
The two demands address dierent types of corpus data and serve different informational aims : Understanding the corpus composition (cf. Section 7.5.1)
 Comparison of corpus queries (cf. Section 7.5.2)

For understanding the corpus composition meta information about
the corpus data and their quantitative distribution are of interest, thus information which can be expressed in categories and numbers. For thecomparison of corpus queries the textual data retrieved from the corpus,
241

including their annotations and distribution over corpus texts are relevant,thus information of higher complexity that cannot be reduced to a nite
set of distinct categories.
Also, for addressing the former demand the visualization should help toget a general understanding of the global corpus data, while for addressing
the latter demand the visualization should allow to compare two sets of local
corpus data. As will be explained in the following two subsections, the rstuse case can be served by adapting existing solutions, while the latter use
case partially requires the development of new solutions.

Functionality: Understanding the corpus composition
A functionality for supporting the understanding of the corpus composition (cf. Section 7.5.1) should aim at presenting information on the corpus
in a clear and easy to grasp form that allows for exploring its dierent aspects. The relevant information for understanding the corpus composition
includes the distinction of the dierent metadata categories39 and token anddocument frequencies for each metadata category. A sample dataset indicating the composition of a hypothetical corpus is given in Table 7.19. Whilethe dierent metadata categories of a corpus usually do not have an inherent
hierarchy40 , they can be arranged hierarchically in response to the users' analytical needs. For example, on the one hand, a user could want to explore
a corpus primarily from a temporal perspective and successively compare
texts of dierent authors within each time period. On the other hand, ananalysis could focalize on comparing dierent authors, with the secondary
objective of analyzing the writings of the authors over time. Summing up,
for the task of understanding the corpus composition we are dealing
with a comparison of frequency information for multiple categories with apossibly hierarchical order.
A basic diagram visualization, like the one presented in Figure 7.1(combined bar and line chart), is suited to display the distribution of texts
and tokens for one metadata dimension at a time. More complex visualizations allow to present multiple data dimensions in one display. In fact,
in IV dierent visualizations for multidimensional or hierarchical data havebeen proposed. For the following discussion we will limit ourselves to three ofthem, which are particularly known and used: (1) Parallel Coordinates (seeAnnex C, Figure C.19), (2) TreeMaps (C.20), and (3) Sunburst diagrams(C.21).
Once we have identied a number of visualizations that are applicableto the type of data we are dealing with (multidimensional data), we haveto consider further aspects of the dataset as well as the informational aims
39

Usually a mixture of categorical (e.g. authors ) and numerical data (e.g. year ).
Some metadata can be hierarchical, for example a publisher can have dierent thematic series with dierent volumes, as shown in the example in Table 7.19.
40

242

Figure 7.19: Example of a corpus composition by metadata243

of the user to choose the most suitable visualization among the pre-selected
options. For example, the number of dimensions and values per dimension
are relevant as well as the hierarchy or order of dimensions and the orderof values within each dimension. Furthermore, the focus of interest canvary among users, thus exible interactions and ltering options have to beconsidered.
The characteristics of the corpus presented in Table 7.19 can be summarized as follows: 6 dierent metadata dimensions
 3 hierarchical dependencies (publisher-series, year-author, lang-author)
 2 to 25 values per dimension, only one dimension with more than 4values

The informational aims related to understanding the corpus composition
can be summarized as follows: overview of the distribution of all dimensions
 ltering of data in relation to dierent dimensions
 analysis of dependencies of the data in relation to specic dimensions

Given the three options, (1) Parallel Coordinates are visualizations for
non-hierarchical data, while (3) TreeMaps and (4) Sunburst diagrams arehierarchical visualizations. In (1) Parallel Coordinates there is no theoreticallimit for the number of dimensions that can be presented and neither forthe number of values per dimension. Comparing dimensions with greatlydierent value ranges will however be less intuitive, and dimensions withan extensive number of values can result in crammed displays. Parallel
Coordinates oer a compact visual presentation of multiple dimensions which
can be dynamically reordered by the user to facilitate the direct comparisonof two dimensions. Also, by means of ltering and color coding single values
can be highlighted and dependencies among values of dierent dimensions
can be brought out. For example, a ltering by publisher A would show thehierarchical dependency between publisher A and series A-1, A-2 and A-3.
Also in (2) TreeMaps and (3) Sunburst diagrams there is no theoreticallimit for the number of dimensions that can be nested within a TreeMap or
Sunburst diagram. However, depending on the size of the display and thenumber of values per dimension, readability will quickly constrain the number of dimensions that are sensible to display at once, with (2) being moresensitive to larger numbers of dimensions than (3). For both visualizations,the user can interactively change the hierarchy of the dimensions by specifying the top level dimension and the order of any subordinate dimensions.
244

Also the number of nested levels to include in the visualization can dynamically be changed. Due to the hierarchical structure of the visualizations
ltering and direct comparison of user-selected dimensions is less straightforward. Filtered elements could be highlighted but the visual hierarchyof the dimensions would make it dicult to grasp quantitative dierencesamong ltered results on lower levels of the hierarchy.
All three visualization types could allow to adapt the unit for measuring
and displaying the quantity of each subcorpus part (e.g. tokens or texts) by
the user in an interactive way.
The visual functionality for understanding the corpus composition
could be integrated with corpus search functionalities, for example by displaying the composition of the results set of a corpus query next to thecorresponding list of concordance lines.

Functionality: Comparison of corpus queries
A functionality for supporting the comparison of query results acrossdierent search cycles (cf. Section 7.5.2) should aim at presenting thequery results in a way that allows to distinguish overlapping and distinct
parts of the query results and to analyze similarities and dierences of thedierent subsets in order to evaluate if all relevant data is covered by thequeries (recall ) or which irrelevant data has been included (precision ). The
relevant information for the comparison of queries includes frequencies ofwords and patterns co-occurring with the search string as well as positionand distributional information.
An example of a query comparison is given in Section 7.5.2, Figure 7.3.
Results of the query (1) 'in the ∗ing' are compared to (2) 'in the _A', thatis the phrase 'in the' followed by a string ending in 'ing' vs. followed by any
adjective ('_A'). The analysis relates to the pairwise comparisons of the threesubsets of results: (1) overlapping with (2), (1) without (2), and (2) without(1). A related functionality should support the comparison of the result sets
in terms of size and in relation to dierent types of information that can bederived from the results sets. The result sets for dierent search cycles related
to one search target are expected to have some overlapping hits and somemutually dierent ones. However, given that the dierent search queries
address a common search target, also the sets of hits that are mutuallydierent are likely to be comparable or structurally related to some extent.
In the following paragraphs dierent levels of information for comparing data
will be described and considerations for their visual implementation will be
discussed.
Quantitative overlap among dierent result sets. Two result sets canshow a full overlap, a subsumption of one set by the other, a partial overlap
with mutually distinct parts, or can be fully distinct with no overlap at all.
Subsumption and partial overlap are the most likely cases. In a rst step, it
245

is important for the user to get an understanding of the share of hits thatshow overlap or are mutually dierent. In a second step, the user can embark
deeper into the analysis and comparison of the hits which are overlappingof distinct from each other. Visualizations can help to provide an easy to
grasp representation of how the two result sets relate quantitatively. For
example, classic set visualizations by Venn diagrams could be used, but also
other graphical representations such as bar charts.
Comparison of dierences in distribution and dispersion of hits.
The distribution of the hits across the distinct result sets can be comparedboth across corpus texts as well as across metadata categories. In both cases,
the discussion of adapted visualization solutions from the previous section
on functionalities for understanding the corpus composition applies.
In addition the dispersion of hits within corpus texts and within sentences
can be relevant. Examples of visual support in the form of dispersion plots
are found in several corpus tools (cf. Table 7.1). Alternatively, dispersion
can also be reduced to a xed set of categorical values ('within rst half ofthe document' vs. 'within second half of the document', in which case again
the visualizations for the distribution in relation to the corpus composition
apply. Unless dispersion is reduced to a xed set of categories, the question ofwhether and how to represent normalization in visualizations for dispersion
data poses a particular challenge, since texts and sentences will be of dierent
lengths.
Comparison of textual patterns and context of the search target.
For complex search queries, which are not equivocal, the user might wantto understand the variation of the search target found in the corpus. In
addition, patterns of the context information surrounding the search hit, forexample within sentence boundaries, are relevant for comparing result sets.
Figure 7.20 gives a sample preview of the information that could be
calculated for the three result sets derived from the comparison of the twoqueries (1) and (2) described above. It includes dierent types of textual andannotation patterns which are identied within a results subset by means of
their frequencies. Since the result sets are most likely of dierent sizes, also
here the question applies of how to deal with normalization.
For the comparison of patterns the linguistic visualizations related to concordances and n-grams (cf. Section 7.6.2) are relevant. They focus on compositional aspects of patterns in one corpus, and more research would be neededto adapt them for the analysis task of comparing patterns across dierent
result sets. A major challenge in comparing patterns of text across result
sets is due to the fact that only a fraction of the patterns found in two result
sets are expected to correspond. This eect increases with the complexityof patterns (e.g., trigrams vs. bigrams) and their positional xedness (e.g.,
directly following the hit vs. somewhere within the sentence). For single
co-occurring words solutions like Parallel Tag Clouds (Collins et al., 2009b)
have been proposed to support comparative analyses, yet more adapted vi246

Figure 7.20: Textual patterns and their frequencies for three result sets
sualizations for comparing patterns of text need to be designed.
Selected examples for each result set. Finally, a variety of dierent
example results for each result set should be presented to the user. To illustrate the idea, Figure 7.3 shows a simple tabular visualization that lists
instantiations of the query patterns based on their occurrence frequencies. In
addition, examples of contextual patterns (e.g. n-grams within the sentence
context) and examples of sentences containing the hit are relevant. The examples should represent a wide spectrum of results which can be determinedby dierent criteria, depending on the users' needs. These could include forexample the variation of search hits, their distribution over dierent corpus
parts (metadata), sentence lengths or sentence complexity, frequent or lessfrequent context elements (semantic), frequent patterns of context (syntactic), etc. However, it is unclear which selection criteria are most useful for
individual users and if they dier between dierent users. Also, the numberof relevant examples can grow large and the best ways for visualizing results
in a compact yet meaningful way still need to be determined.
Summing up, for the task of comparing corpus queries we are dealing with dierent types of data. On the one hand we are concerned witha quantitative comparison of dierent subsets of the results. On the other
hand, we also focus on the comparison of categorical (i.e., metadata, positional information) and nominal data (i.e., co-occurring words, n-grams)
and related frequency information. These frequency information are related
to the individual result sets of potentially dierent size and are thus not247directly comparable without any kind of normalization. While powerful visualizations for quantitative and categorical information exist and some visualizations are available for positional and distributional information, thevisualization of textual data in the form of sentence examples or patterns of
linguistic units within sentences are still very limited. To develop powerfulvisualization-based functionalities for the comparison of result sets to dierent but related corpus queries, we need to both identify the most relevant
aspects of the data and develop visualizations that can bring them out in aclear and concise form.
7.7

Conclusions

In this chapter the open demands derived from the user studies were discussed within the bigger picture of the corpus linguistics that is embedded
both in the linguistics research context as well as the surrounding computational developments and initiatives. It could be shown that dierent demands call for solutions on dierent practical levels, of which the extension
and advancement of functionalities is the peak of the challenge.
By providing a brief excursus into the domain of information visualizationand its targeted eorts towards the representation of linguistic and language
content, a promising path for extending the arsenal of corpus tool functionalities was highlighted and relevant considerations for its pursuit werediscussed in relation to two selected corpus tool demands.

248

-

Chapter8-

Conclusions

249

250

Conclusions

This research aimed to investigate on open demands in corpus research practice and related needs for tools support. Based on the results from interview
and survey investigations with more than 100 corpus users, it can be concluded that the discipline is indeed facing a wide range of open demands
on dierent operational levels, and of varying granularity. The results indicate that, other than expected, the demands are more prominent for issues
related to the quality and availability of resources, the usability of tools,
and challenges in applying corpus research methods, than for functional demands related to corpus analyses. While the user investigations yield anumber of concrete requirements related to existing tool functionalities, demands related to broader linguistic analysis needs and related innovative
tool functionalities remain abstract and are rather part of an improved understanding of the corpus usage situation than a set of clear indications forready-to-implement new functionalities.
This situation is in line with known challenges from user-centered research on open-ended subjects and tools for their support, as discussed in
the methodology parts of this thesis. Given the experienced diculty to collect explicit and wide-reaching demands directly from corpus users, alreadyin the rst parts of the user study, the research of this thesis focused on understanding and describing the corpus usage situation in general, by investigating on workows, theoretical principles and analysis activities throughenquiries with corpus users. This included to identify the building blocks ofcorpus research practice and to dene a framework of reference in the form of
selecting relevant subjects, questions and answer options for the structured
investigations. Indeed, in the literature the groundwork for the research ofthis thesis was mostly missing, both in terms of a systematic description of
the corpus research scenario, and in terms of substantial related works onuser-centered studies in corpus linguistics.
Finally, the research of this thesis makes two main contributions for advancing user-centered studies and tools support.
On the one hand, it provides a systematic description of the building
blocks of the corpus research situation and thus states a framework of reference for corpus user studies. Also, it oers a review of approaches for usercentered research in general, and of particular challenges related to openended research as well as solutions that have been proposed to counter thechallenges.
On the other hand, it determines the scope and variety of demands in corpus research. It presents a comprehensive list of the identied open demands
and a grouping of them along four thematic lines, related to (1) corpus resources, (2) general aspects of tools, (3) corpus analysis procedures, and (4)best practices. Furthermore, it discusses dierent strategies for responding251

to the demands. The strategies relate to measures on the level of (1) documentation and training, (2) technical and usability adaptations, (3) workow
and data management, and (4) processing and analysis functionalities.
In relation to the functional demands for corpus tools, the core interestof this research, the discussion of potential responses to the open demands istaken one step further. First ideas of how to design advanced functionalities
are proposed and motivated. Together with the discussion of general strategies for responding to the open demands, the depicted example solutions
provide a roadmap for how abstract open demands related to generic corpus
usage situations can be approached.
As the presented research to our knowledge is the rst of a kind to approach a user-centered study on corpus research needs with a larger andgeneral scope, the presented results open up several opportunities for further research.
First, the theoretical framework stipulated through the description of thecentral corpus building blocks and related categories provides a basis for further theoretical works on dening corpus research practice and establishingsystematic and measurable categories for its description and evaluation.
Second, based on the grouping of open demands, for which partially solutions exist, further research should be carried out to carefully map specicdemands to any existing works that provide partial solutions. This would
allow to bring together relevant knowledge and to identify which aspects ofa demand are still unresponded to and need further address.
Third, to better understand the potential of the proposed solutions andideas for functional extensions, the suggestions could be taken as starting
points to design and implement new functionalities and to evaluate their
use through user tests. In the same way, the overview and outline of how
to approach the use of advanced visualization techniques, could be used todevelop extended visual support functionalities and to evaluate them basedon user testing.
To sum it up, the presented research started the endeavor to bring the
corpus user into the loop of advancing corpus tool functionalities. It reachedto the point where a context and rst insights for describing the practical
corpus usage situation and related demands has been created, and indications
for how to approach and rene the identied demands have been discussed.
The delivered research opens up several paths for practical and theoreticalfollow-up works for extending the range of functionalities for corpus analysis
tools.

252

Bibliography

K. Aijmer. Corpora and Language Teaching, volume 33 of Studies in Corpus Linguistics.
John Benjamins Publishing Company, 2009. ISBN
9789027289988.
J. C. Alderson. Judging the frequency of english words. Applied Linguistics,
28(3):383409, 2007. doi: 10.1093/applin/amm024. URL +http://dx.
doi.org/10.1093/applin/amm024.
N. K. Anagnostou and G. R. S. Weir. Review of software applications forderiving collocations. In ICT in the Analysis, Teaching and Learning of
Languages, Preprints of the ICTATLL Workshop 2006, pages 91100, August 2006. URL https://strathprints.strath.ac.uk/2383/.
G. Andersen. How to use corpus linguistics in sociolinguistics, pages 547
562. Routledge, 2010.
T. Anderson, A. Hussam, B. Plummer, and N. Jacobs. Pie charts for visualizing query term frequency in search results. In ICADL, 2002.
D. Angus, A. E. Smith, and J. Wiles. Conceptual recurrence plots: Revealing
patterns in human discourse. IEEE Trans. Vis. Comput. Graph., 18(6):988997, 2012. doi: 10.1109/TVCG.2011.100. URL https://doi.org/
10.1109/TVCG.2011.100.
L. Anthony. Issues in the design and development of software tools for corpus
studies: The case for collaboration., chapter 6, pages 87104. Continuum
International Publishing Group, London, UK, 2009.
L. Anthony. Developing antconc for a new generation of corpus linguists. In
Proceedings of the Corpus Linguistics conference (CL 2013), pages 1416,
2013a.
L. Anthony. A critical look at software tools in corpus linguistics. Linguistic
Research, 30(2):141161, 2013b. ISSN 1229-1374. doi: 10.17250/khisli.30.
2.201308.001.
253

L. Anthony. Antconc (version 3.5.9) [computer software], 2020. URL https://www.laurenceanthony.net/software.
O. Ari. Review of three software programs designed to identify lexical bundles. Language Learning and Technology, 10(1):3037, 2006.
M. Baker, G. Francis, and E. Tognini-Bonelli. 'Corpus Linguistics and Translation Studies: Implications and Applications'. John Benjamins PublishingCo., Netherlands, 1993.
P. Baker. Contemporary Corpus Linguistics. Continuum International Publishing Group, London, UK, 2009.
M. Barlow. MonoConc Pro. Athelstan, Houston, TX, 2000.
M. Baroni and S. Bernardini. Bootcat: Bootstrapping corpora and terms
from the web. In In Proceedings of LREC 2004, pages 13131316, 2004.
M. Beiÿwenger and A. Storrer. Corpora of Computer-Mediated Communication, volume 1 of Handbooks of Linguistics and Communication Science
29.1, pages 292308. Mouton de Gruyter, Berlin, New York, 2008.
H. Bergenholtz and S. Tarp. Die moderne lexikographische funktionslehre:
Diskussionsbeitrag zu neuen und alten paradigmen, die wörterbücher als
gebrauchsgegenstände verstehen. Lexicographica, 18/2002:253263, 2003.
ISSN 0175-6206.
H. Bergenholtz and S. Tarp. Wörterbuchfunktionen, pages 1125. Finnische
Beiträge zur Germanistik. Peter Lang, 2005. ISBN 3631545320.
H. Bergenholtz, S. Tarp, and G. Duvå.

Manual of Specialised Lexicog-

Benjamins TransISBN 9789027216120. URL
https://books.google.it/books?id=6E1uHygDwQwC.
raphy:The Preparation of Specialised Dictionaries.

lation Library 12. J. Benjamins, 1995.

J. Bertin. Semiology of Graphics. University of Wisconsin Press, 1983. ISBN
0299090604.
D. Biber. Variation across Speech and Writing. Cambridge University Press,
Cambridge, 1988.
D. Biber. What can a corpus tell us about registers and genres?, pages 241
254. Routledge, 2010.
D. Biber, S. Conrad, and R. Reppen. Corpus linguistics : investigating
language structure and use. Cambridge University Press, Cambridge [u.a.],
1998.
254

A. Bogner, B. Littig, and W. Menz. Das Experteninterview. Theorie, Methode, Anwendung. VS, Verlag für Sozialwissenschaften, 2. auage edition,
2005.
J. Bortz and N. Döring. Forschungsmethoden und Evaluation. SpringerLehrbuch. Springer-Medizin-Verl., Heidelberg, 4., überarb. au. edition,
2006.
T. J. Bouchard. Field research methods: interviewing, questionnaires, participant observation, systematic observation, unobtrusive measures. In M. D.
Dunnette, editor, Handbook of Industrial and Organizational Psychology.
Rand McNally, Chicago, 1976.
A. Boulton. Beyond concordancing: Multiple aordances of corpora in university language degrees. Procedia - Social and Behavioral Sciences, 34:
3338, 2012. URL https://hal.archives-ouvertes.fr/hal-00502633.
A. Boulton. Separating fact and ction: The real story of corpus use inlanguage teaching. In L. Bradley and S. Thouesny, editors, 20 Yearsof EUROCALL: Learning from the Past, Looking to the Future. Proceedings of the 2013 EUROCALL Conference, pages 5156. Dublin/Voillans:

Research-publishing.net, 2013. doi: 10.14705/rpnet.2013.000138.

V. Brezina and team. #lancsbox v. 5: A new-generation corpus analysis tool
[webinar], 2020. URL https://www.youtube.com/watch?v=lA6gjoXGB20.
V. Brezina, P. Weill-Tessier, and A. McEnery. #lancsbox v. 5.0. [software],2020. URL http://corpora.lancs.ac.uk/lancsbox.
H. Brugman and A. Russel. Annotating multimedia/ multi-modal resources
with elan. In In Proceedings of LREC 2004, Fourth International Conference on Language Resources and Evaluation., 2004.
A. Burchardt, K. Erk, A. Frank, A. Kowalski, and S. Pado. SALTO: A
versatile multi-level annotation tool. In Proceedings of LREC-2006, Genoa,
Italy, 2006.
M. Butt, A. Hautli-Janisz, and V. Lyding. LingVis: Visual Analytics for
Linguistics. CSLI Publications, Stanford, 2020.
N. Cao, J. Sun, Y.-R. Lin, D. Gotz, S. Liu, and H. Qu. Facetatlas: Multifaceted visualization for rich text corpora. Visualization and Computer Graphics, IEEE Transactions on, 16:1172  1181, 12 2010. doi:
10.1109/TVCG.2010.154.
S. K. Card, , B. Shneiderman, and J. D. MacKinlay. Readings in Information
VisualizationUsing Vision to Think. Series in Interactive Technologies.
Morgan Kaufmann Publishers, 1st ed. edition, 1999. ISBN 1-55860-533-9.
255

M. S. T. Carpendale. Considering visual variables as a basis for information visualisation, 2003. URL https://prism.ucalgary.ca/handle/
1880/45758.
K.-U. Carstensen, C. Ebert, C. Ebert, S. Jekat, R. Klabunde, and H. Langer,
editors. Computerlinguistik und Sprachtechnologie: Eine Einführung.
Spektrum, Heidelberg, 3 edition, 2009.
M. Chen, S.-T. Huang, T.-H. Kao, H.-w. Chiu, and T.-H. Yen. Glance visualizes lexical phenomena for language learning. In Proceedings of theWorkshop on Interactive Language Learning, Visualization, and Inter-faces, pages 3437, Baltimore, Maryland, USA, June 2014. Association for

Computational Linguistics. URL http://www.aclweb.org/anthology/
W14-3105.

W. Cheng, C. Greaves, and M. Warren. From n-gram to skipgram to concgram. Corpus Linguistics, 11(4):411433, 2006.
C. Chiarcos, S. Dipper, M. Götze, U. Leser, A. Lüdeling, J. Ritz, and
M. Stede. A exible framework for integrating annotations from dierent
tools and tag sets. TRAITEMENT AUTOMATIQUE DES LANGUES.
VOLUME 49  No 2/2008, PAGES 217 a 246, 2008.
N. Chomsky. A transformational approach to syntax. In A. A. Hill, editor, Proceedings of the Third Texas Conference on Problems of Linguistic
Analysis in English, May 1958, pages 124148, Austin, 1962.
N. Chomsky. Aspects of the Theory of Syntax. The MIT Press, Cambridge,
MA, 1965.
N. Chomsky. Language and mind. Harcourt, Brace & world. Harcourt, Brace& World, 1968.
O. Christ. A modular and exible architecture for an integrated corpus query
system. In Proceedings of COMPLEX'94: 3rd Conference on Computational Lexicography and Text Research, pages 2332, Budapest, Hungary,
1994.
K. W. Church and R. L. Mercer. Introduction to the special issue on computational linguistics using large corpora. Comput. Linguist., 19(1):124,
Mar. 1993. ISSN 0891-2017.
C. Collins, S. Carpendale, and G. Penn. Docuburst: Visualizing document
content using language structure. Computer Graphics Forum (Proc. of
the Eurographics/IEEE-VGTC Symposium on Visualization (EuroVis)),28(3):1039  1046, 2009a. doi: 10.1111/j.1467-8659.2009.01439.x.
256

C. Collins, F. Viegas, and M. Wattenberg. Parallel tag clouds to explore andanalyze faceted text corpora. In Visual Analytics Science and Technology,
2009. VAST 2009. IEEE Symposium on, pages 9198, Oct 2009b. doi:
10.1109/VAST.2009.5333443.
S. Conrad. What can a corpus tell us about grammar?, pages 227240. Routledge, 2010. doi: 10.4324/9780203856949.ch17.
W. Cui, H. Qu, H. Zhou, W. Zhang, and S. Skiena. Watch the story unfold
with textwheel: Visualization of large-scale news streams. ACM Trans. Intell. Syst. Technol., 3(2):20:120:17, 2012. doi: 10.1145/2089094.2089096.
URL https://doi.org/10.1145/2089094.2089096.
C. Culy and V. Lyding. Double tree: An advanced kwic visualization forexpert users. In Information Visualisation (IV), 2010 14th International
Conference, pages 98103, July 2010. doi: 10.1109/IV.2010.24.
C. Culy and V. Lyding. Corpus Clouds - Facilitating Text Analysis by Meansof Visualizations, pages 351360. Lecture Notes in Computer Science.
Springer, Berlin, Heidelberg, 2011.
C. Culy, V. Lyding, and H. Dittmann. xldd: Extended linguistic dependency
diagrams. In E. Banissi, S. Bertschi, R. A. Burkhard, U. Cvek, M. J.
Eppler, C. Forsell, G. G. Grinstein, J. Johansson, S. Kenderdine, F. T.
Marchese, C. Maple, M. Trutschl, M. Sarfraz, L. J. Stuart, A. Ursyn,
and T. G. Wyeld, editors, 15th International Conference on Information
Visualisation, IV 2011, London, United Kingdom, July 13-15, 2011, pages
164169. IEEE Computer Society, 2011. doi: 10.1109/IV.2011.42. URL
https://doi.org/10.1109/IV.2011.42.
A. Curzan. Historical corpus linguistics and evidence of language change,
volume 2 of Handbooks of Linguistics and Communication Science 29.2,
pages 10911109. Mouton de Gruyter, Berlin, New York, 2008.
A. Damascelli and A. Martelli.

Corpus Linguistics and Computational

Linguistics: An Overview with Special Reference to English.

Celid perl'Università. Celid, 2003. ISBN 9788876615368. URL http://books.
google.it/books?id=OsYYAAAACAAJ.

J. de Does, J. Niestadt, and K. Depuydt. Creating Research Environmentswith BlackLab, pages 245257. London:Ubiquity, 2017. URL https://doi.org/10.5334/bbi.20.
P. DeCamp, A. Frid-Jimenez, J. Guiness, and D. Roy. Gist icons: Seeing
meaning in large bodies of literature, 2015.
257

D. Dela£, Z. Krleºa, J. najder, B. Dalbelo Ba²i¢, and F. ari¢. TermeX: A Tool for Collocation Extraction, pages 149157. Springer Berlin
Heidelberg, Berlin, Heidelberg, 2009. URL https://doi.org/10.1007/
978-3-642-00382-0_12.
F. Dias and R. Minghim. xhipp: extended hierarchical point placement
strategy. 2018 31st SIBGRAPI Conference on Graphics, Patterns andImages (SIBGRAPI), pages 361368, 2018.
H. Diessel.

Corpus linguistics and rst language acquisition, volume 2 of

Handbooks of Linguistics and Communication Science 29.2, pages 1197

1212. Mouton de Gruyter, Berlin, New York, 2008.

N. Diewald, M. Hanl, E. Margaretha, J. Bingel, M. Kupietz, P. Ba«ski, and
A. Witt. KorAP architecture  diving in the deep sea of corpus data. In
Proceedings of the Tenth International Conference on Language Resources
and Evaluation (LREC'16),

pages 35863591, Portoroº, Slovenia, May
2016. European Language Resources Association (ELRA). URL https://www.aclweb.org/anthology/L16-1569.
A. Dillon. Usability evaluation. Taylor and Francis, London, 2001.
L. Diniz. Comparative review: Textstst 2.5, antconc 3.0, and compleat lexical
tutor 4.0. Language Learning and Technology, 9(3):2227, 2005.
S. Dipper. Xml-based stand-o representation and exploitation of multi-level
linguistic annotation. In In Proc. of Berliner XML Tage 2005 (BXML
2005), pages 3950, Berlin, Germany, 2005.
H. Dittmann, M. Dur£co, A. Geyken, T. Roth, and K. Zimmer. Korpus C4 
a distributed corpus of German varieties. Number 14 in Hamburg Studies
on Multilingualism. Amsterdam: Benjamins, 2012.
A. Don, E. Zheleva, M. Gregory, S. Tarkan, L. Auvil, T. Clement, B. Shneiderman, and C. Plaisant. Discovering interesting usage patterns in text
collections: Integrating text mining with visualization. In Proceedings of
the Sixteenth ACM Conference on Conference on Information and Knowl-edge Management, CIKM '07, page 213222, New York, NY, USA, 2007.

ACM, ACM. ISBN 978-1-59593-803-9. doi: 10.1145/1321440.1321473.
URL http://doi.acm.org/10.1145/1321440.1321473. 00107.
R. Eckart de Castilho and I. Gurevych. A broad-coverage collectionof portable NLP components for building shareable analysis pipelines.
In Proceedings of the Workshop on Open Infrastructures and Analysis
Frameworks for HLT, pages 111, Dublin, Ireland, Aug. 2014. Association for Computational Linguistics and Dublin City University. doi:
10.3115/v1/W14-5201. URL https://aclanthology.org/W14-5201.
258

R. Eckart de Castilho, C. Biemann, I. Gurevych, and S. M. Yimam. Webanno: a exible, web-based annotation tool for clarin. In Proceedingsof the CLARIN Annual Conference (CAC) 2014, page online, Utrecht,
Netherlands, Oct. 2014. CLARIN ERIC. Extended abstract.
J. Egbert, T. Larsson, and D. Biber.

Doing Linguistics with a Corpus:

Methodological Considerations for the Everyday User (Elements in Corpus
Linguistics). Cambridge University Press, Cambridge, 2020. URL https://doi.org/10.1017/9781108888790.

S. Evert and A. Hardie. Twenty-rst century corpus workbench: Updating aquery architecture for the new millennium. In In Proceedings of the Corpus
Linguistics 2011 conference, University of Birmingham, UK, 2011.
C. Fantinuoli and F. Zanettin. New directions in corpus-based translation studies.
Language Science Press, Berlin, 2015.
ISBN
978-3-944675-83-1.
URL http://www.diss.fu-berlin.de/docs/
servlets/MCRFileNodeServlet/FUDOCS{_}derivate{_}000000005022/
76-3-287-1-10-20150603.pdf.

C. J. Fillmore. ``Corpus linguistics'' vs. ``computer-aided armchair linguistics''. In Directions in Corpus Linguistics: Proceedings from a 1991 Nobel
Symposium on Corpus Linguistics, pages 3566, Stockholm, 1992. Mouton
de Gruyter, Mouton de Gruyter.
J. Firth. Papers in linguistics, 1934-1951. Oxford University Press, Oxford,
1957.
F. Fowler. Improving Survey Questions: Design and Evaluation. AppliedSocial Research Methods. SAGE Publications, California, 1995. ISBN
9780803945838.
W. N. Francis and H. Ku£era. Manual of information to accompany 'a standard corpus of present-day edited american english, for use with digitalcomputers with digital computers'. Technical report, Department of Linguistics, Brown University, Providence, RI, 1964.
A. Frankenberg-Garcia. A peek into what today's language learners as researchers actually do. International Journal of Lexicography, 18(3):335
355, 2005. doi: 10.1093/ijl/eci015. URL +http://dx.doi.org/10.1093/
ijl/eci015.
D. Freedman, R. Pisani, and R. Purves. Statistics. Norton, New York, 4thedition edition, 2007.
S. Friese. Qualitative Data Analysis with ATLAS.ti. SAGE Publications,2011. ISBN 9781446253717. URL https://books.google.it/books?id=
p4Pnyi9-yb0C.
259

P. Fuertes-Olivera and H. Bergenholtz.

net, Digital Initiatives and Lexicography.

ISBN 9781441128065.
E2UTCiTYJpsC.

E-Lexicography:

The Inter-

Continuum, London, 2011.

URL https://books.google.it/books?id=

M. Gärtner, U. Hahn, and S. Hermann. Preserving workow reproducibility: The RePlay-DH client as a tool for process documentation. In Proceedings of the Eleventh International Conference on Language Resources

and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Lan-guage Resources Association (ELRA). URL https://aclanthology.org/
L18-1089.

M. Gatto. The Web As Corpus: Theory and Practice. Corpus and Discourse.
Bloomsbury Academic, 2014. ISBN 9781441150981. URL http://books.
google.it/books?id=SixrmAEACAAJ.
G. Gilquin and S. T. Gries. Corpora and experimental methods: a stateof-the-art review. Corpus Linguistics and Linguistic Theory, 5(1):126,
2009.
J. Gläser and G. Laudel. Experteninterviews und qualitative Inhaltsanalyse
als Instrumente rekonstruierender Untersuchungen. Lehrbuch. VS, Verl.
für Sozialwiss., Wiesbaden, 3., überarb. au. edition, 2009.
A. Gomide Rodrigues.

Corpus Linguistics Software:

understanding theirusages and delivering two new tools. PhD thesis, Lancaster University,
Department of Linguistics and English Language, 2020.

M. Good. Software usability engineering. Digital Technical Journal, 1(6):125133, Feb 1988.
C. Görg, Z. Liu, J. Kihm, J. Choo, H. Park, and J. T. Stasko. Combining computational analyses and interactive visualization for document exploration and sensemaking in jigsaw. IEEE Trans. Vis. Comput. Graph., 19(10):16461663, 2013. doi: 10.1109/TVCG.2012.324. URL
http://doi.ieeecomputersociety.org/10.1109/TVCG.2012.324.
S. Granger, G. Gilquin, and F. Meunier.
Corpus Research. Looking Back,

Twenty Years of LearnerMoving Ahead:

Proceedings of theFirst Learner Corpus Research Conference (LCR 2011).

Corpora andLanguage in Use - Proceedings. Presses universitaires de Louvain,
2013. ISBN 9782875581990. URL http://books.google.it/books?id=
2xBeAgAAQBAJ.
M. Granitzer, W. Kienreich, V. Sabol, K. Andrews, and W. Klieber. Evaluating a system for interactive exploration of large, hierarchically structured
document repositories. IEEE Symposium on Information Visualization,
pages 127134, 2004.
260

S. Greenbaum. Comparing English Worldwide: The International Corpus ofEnglish. Clarendon Press, Oxford, 1996.
S. T. Gries. What is corpus linguistics? Language and Linguistics Compass,3(5):12251241, 2009. URL http://dblp.uni-trier.de/db/journals/
llc/llc3.html#Gries09.
S. T. Gries. Corpus linguistics and theoretical linguistics: a love-hate relationship? not necessarily . . . . International Journal of Corpus Linguistics,
15(3):327343, 2010.
J. T. Hackos and J. C. Redish. User and Task Analysis for Interface Design.
John Wiley & Sons, Inc., New York, NY, USA, 1998. ISBN 0-471-17831-4.
K.-C. Hamborg and G. Gediga. Ingenieurpsychologie. enzyklopädie der psychologie d/iii/2. chapter Methoden und Modelle für die Gestaltung gebrauchstauglicher Software, pages 495529. Hogrefe, Göttingen, 2006.
A. Hardie. Cqpweb - combining power, exibility and usability in a corpus
analysis tool. International Journal of Corpus Linguistics, 17(3):380409, 2012. ISSN 1384-6655. doi: 10.1075/ijcl.17.3.04har.
A. Hardie and T. McEnery. On two traditions in corpus linguistics, and what
they have in common. International Journal of Corpus Linguistics, 15(3):384394, 2010.
M. Hassenzahl. User experience (ux): towards an experiential perspective
on product quality. In E. Brangier, G. Michel, J. M. C. Bastien, and
N. Carbonell, editors, IHM, ACM International Conference Proceeding
Series, pages 1115. ACM, 2008. URL http://dblp.uni-trier.de/db/
conf/ihm/ihm2008.html#Hassenzahl08.
S. Havre, B. Hetzler, and L. Nowell. Themeriver: visualizing theme changes
over time. In Information Visualization, 2000. InfoVis 2000. IEEE Symposium on, pages 115123, 2000. doi: 10.1109/INFVIS.2000.885098.
M. A. Hearst. Search User Interfaces. Cambridge University Press, New
York, NY, USA, 1st edition, 2009. ISBN 0521113792, 9780521113793.
U. Heid. Electronic dictionaries as tools: towards an assessment of usability.
Continuum, London, 2011.
C. Helerich. Die Qualität qualitativer Daten: Manual für die Durchführung
qualitativer Interviews. Lehrbuch: VS Verl. für Sozialwissenschaften. VS,
Verlag für Sozialwissenschaften, Wiesbaden, 2. auage edition, 2005.
A. Herold and K. Stathi. Measuring syntagmatic xedness of multi-word expressions. In Proceedings of the Corpus Linguistics Conference (CL 2007),
pages 117, University of Birmingham, UK, 07 2007.
261

E. W. Hinrichs, M. Hinrichs, and T. Zastrow. WebLicht: Web-Based LRT
Services for German. In Proceedings of the ACL 2010 System Demonstrations, pages 2529, 2010. URL http://www.aclweb.org/anthology/
P10-4005.
O. Hoeber and X. D. Yang. User-oriented evaluation methods for interactive web search interfaces. In Proceedings of the 2007 IEEE/WIC/ACM
International Conferences on Web Intelligence and Intelligent Agent Technology - Workshops,

WI-IATW '07, pages 239243, Washington, DC,
USA, 2007. IEEE Computer Society. ISBN 0-7695-3028-1. URL http://dl.acm.org/citation.cfm?id=1339264.1339699.

S. Homann and S. Evert. BNCweb (CQP edition) - the marriage of two
corpus tools., pages 177195. Peter Lang, 2006.
E. Hoque, G. Carenini, and S. Joty. Interactive exploration of asynchronous
conversations: Applying a user-centered approach to design a visual text
analytic system. In Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces, pages 4552, Baltimore,
Maryland, USA, June 2014. Association for Computational Linguistics.
doi: 10.3115/v1/W14-3107. URL https://www.aclweb.org/anthology/
W14-3107.
S. Hunston. Corpora in Applied Linguistics. Cambridge applied linguistics
series. Cambridge University Press, Cambridge, 2002.
G. Iarossi. The Power of Survey Design. The World Bank, Washington ,D.C., 2006.
ISO-9241-11. ISO 9241-11: Ergonomic Requirements for Oce Work with
Visual Display Terminals (VDTs): Part 11: Guidance on Usability. International Organization For Standardization, 1998. URL https://books.
google.it/books?id=TzXYZwEACAAJ.
S. Johansson, G. Leech, and H. Goodluck. Manual of information to accompany the lancaster-olso/bergen corpus of british english, for use withdigital computers. Technical report, Department of English, Oslo University, Oslo, 1978.
P. W. Jordan. An Introduction To Usability. CRC, 1 edition, 1998.
D. Jurafsky and J. H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, andSpeech Recognition.

Prentice Hall PTR, Upper Saddle River, NJ, USA,
1st edition, 2000. ISBN 0130950696.
262

C. Kennedy and T. Miceli. An evaluation of intermediate students' approaches to corpus investigation. Language Learning and Technology, 5(3):7790, 2001. URL http://llt.msu.edu/vol5num3/kennedymiceli/.
G. Kennedy. An Introduction to Corpus Linguistics. Longman, London, New
York, 1998.
A. Kerren, A. Ebert, and J. Meyer, editors. Human-centered Visualization
Environments. Springer-Verlag, Berlin, Heidelberg, 2007. ISBN 3-54071948-2, 978-3-540-71948-9.
D. Kiesel, P. Riehmann, H. Wachsmuth, B. Stein, and B. Fröhlich. Visual
analysis of argumentation in essays. IEEE Transactions of Visualization
and Computer Graphics, 2020.
A. Kilgarri. Sketching words. Technical Report ITRI-02-17, Information
Technology Research Institute, University of Brighton, 2002. Also published in "Lexicography and Natural Language Processing: A Festschriftin Honour of B.T.S. Atkins".
A. Kilgarri, P. Rychlý, P. Smrº, and D. Tugwell. A modular and exible
architecture for an integrated corpus query system. In Proceedings of the
11th EURALEX International Congress, page 105116, Bretagne, France,
2004.
A. Kilgarri, V. Baisa, J. Bu²ta, M. Jakubí£ek, V. Kovvá°, J. Michelfeit,
P. Rychlý, and V. Suchomel. The sketch engine: ten years on. Lexicography, pages 736, 2014.
E. König and W. Lezius. The tiger language - a description language for syntax graphs, formal denition. Technical report, IMS, Universität Stuttgart,
Germany, 2003.
D. Knight, F. Loizides, S. Neale, L. Anthony, and I. Spasi¢. Developing
computational infrastructure for the corcencc corpus: The national corpus
of contemporary welsh. Language Resources and Evaluation, 2020. doi: 10.
1007/s10579-020-09501-9. URL https://link.springer.com/article/
10.1007%2Fs10579-020-09501-9.
L. Kocincová, V. Baisa, M. Jakubí£ek, and V. Kovár. Interactive visualizations of corpus data in sketch engine. In Proceedings of the Workshop on
Innovative Corpus Query and Visualization Tools at NODALIDA 2015,
May 11-13, 2015, Vilnius, Lithuania, pages 1722, Linköping, Sweden,

May 2015. Linköping University Electronic Press.

R. Kosara. The cost of a sick chart, 2009. URL https://eagereyes.org/
criticism/cost-of-a-sick-chart.
263

T. Krause and A. Zeldes. Annis3: A new architecture for generic corpus
query and visualization. Digital Scholarship in the Humanities, 31(1):118139, 2016. doi: 10.1093/llc/fqu057. URL +http://dx.doi.org/10.
1093/llc/fqu057.
K. Kucher and A. Kerren. Text visualization techniques: Taxonomy,
visual survey, and community insights. In 2015 IEEE Pacic Visualization Symposium (PacicVis), pages 117121, Los Alamitos, CA,
USA, apr 2015. IEEE Computer Society. doi: 10.1109/PACIFICVIS.
2015.7156366. URL https://doi.ieeecomputersociety.org/10.1109/
PACIFICVIS.2015.7156366.
O. A. Kulyk, R. Kosara, J. Urquiza-Fuentes, and I. H. C. Wassink. Humancentered aspects. In A. Kerren, A. Ebert, and J. Meyer, editors, HumanCentered Visualization Environments, volume 4417 of Lecture Notes in
Computer Science, pages 1375. Springer, 2006.
URL http://dblp.
uni-trier.de/db/conf/dagstuhl/hcve2006.html#KulykKUW06.
G. Leech. The state of the art in corpus linguistics. In K. Aijmer and
B. Altenberg, editors, English Corpus Linguistics: Studies in Honour of
Jan Svartvik., pages 829. Longman, London, 1991.
L. Lemnitzer and H. Zinsmeister. Korpuslinguistik eine Einführung. NarrStudienbücher. Narr, Tübingen, 2 edition, 2010. ISBN 9783823365556.
L. Leventhal and J. Barnes. Usability Engineering: Process, Products, andExamples. Pearson/Prentice Hall, 2008. ISBN 9780131570085. URL
https://books.google.it/books?id=3gglAQAAIAAJ.
V. Lyding and C. Culy. Visualizing linguistic data: From principles to toolkits for doing it yourself, 2012. URL https://studylib.net/doc/5800622.
V. Lyding, L. Nicolas, and E. Stemle. 'interhist' ì¶ an interactive visualinterface for corpus exploration. In N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, and
S. Piperidis, editors, Proceedings of the Ninth International Conferenceon Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland,
May 26-31, 2014,

pages 635641. European Language Resources Association (ELRA), 2014. URL http://www.lrec-conf.org/proceedings/
lrec2014/summaries/517.html.
J. T. MacClave, T. Sinich, and T. L. Sincich. Statistics. Prentice Hall, Upper
Saddle River, NJ, 8th edition edition, 2000.
T. Machálek. KonText: Advanced and exible corpus query interface.
In Proceedings of the 12th Language Resources and Evaluation Conference, pages 70037008, Marseille, France, May 2020. European Lan264

guage Resources Association. ISBN 979-10-95546-34-4. URL https://www.aclweb.org/anthology/2020.lrec-1.865.
B. MacWhinney and C. Snow. The child language data exchange system. Journal of Child Language, 12(2):271295, 1985. doi: 10.1017/
S0305000900006449.
S. S. Mangiaco. Summary and analysis of extension program evaluation inr, version 1.18.8., 2016. URL rcompanion.org/handbook/.
C. D. Manning and H. Schütze. Foundations of Statistical Natural Language
Processing. MIT Press, Cambridge, MA, USA, 1999. ISBN 0-262-13360-1.
O. Mason. Developing software for corpus research. International Journalof English Studies, 8(1):141156, Jun. 2008. URL https://revistas.um.
es/ijes/article/view/49141.
T. Mayer, C. Rohrdantz, M. Butt, F. Plank, and D. Keim. Visualizing vowel
harmony. Linguistic issues in language technology, 4(2):133, 2010.
P. Mayring. Qualitative Inhaltsanalyse. UTB für Wissenschaft. Beltz, Weinheim und Basel, 8. auage edition, 2003. ISBN 9783825282295. URL
https://books.google.it/books?id=Q-imSAAACAAJ.
T. McEnery and A. Hardie. Corpus Linguistics: Method, Theory and Practice. Cambridge Textbooks in Linguistics. Cambridge University Press,2012. doi: 10.1017/CBO9780511981395.
T. McEnery and A. Wilson. Corpus linguistics : an introduction. Edinburgh
Univ. Press, Edinburgh, 2005.
P. Meurer, V. Rosén, and K. D. Smedt. Interactive visualizations in
the iness treebanking infrastructure. In A. Hautli-Janisz and V. Lyding, editors, Proceedings of the LREC 2016 Workshop 'VisLR II: Visualization as Added Value in the Development, Use and Evaluation of
Language Resources', LREC 2016, Portoroz, Slovenia, May 23, 2016,

pages 17. European Language Resources Association (ELRA), 2016.
URL http://www.lrec-conf.org/proceedings/lrec2016/workshops/
LREC2016Workshop-VisLR%20II_Proceedings.pdf.
C. Müller. Mmax2 annotation schemes (draft). Technical report, EML Research gGmbH, 2005. URL http://mmax.eml-research.de.
C. Müller and M. Strube. Multi-level annotation of linguistic data with
MMAX2. In S. Braun, K. Kohn, and J. Mukherjee, editors, Corpus Technology and Language Pedagogy: New Resources, New Tools, New Methods,
pages 197214. Peter Lang, Frankfurt a.M., Germany, 2006.
265

J. Nielsen. Usability Engineering. Morgan Kaufmann Publishers Inc., San
Francisco, CA, USA, 1993. ISBN 0125184050.
J. Nielsen. Usability inspection methods. chapter Heuristic Evaluation,
pages 2562. John Wiley & Sons, Inc., New York, NY, USA, 1994. ISBN
0-471-01877-5.
B. O'Connor. MiTextExplorer: Linked brushing and mutual informationfor exploratory text data analysis. In Proceedings of the Workshop on
Interactive Language Learning, Visualization, and Interfaces, pages 113,
Baltimore, Maryland, USA, June 2014. Association for Computational
Linguistics. doi: 10.3115/v1/W14-3101. URL https://www.aclweb.org/
anthology/W14-3101.
A. O'Keee and M. McCarthy, editors. The Routledge Handbook of Corpus
Linguistics. Routledge, 2010. doi: 10.4324/9780203856949.
A. O'Keee, M. McCarthy, and R. Carter. From Corpus to Classroom:
Language Use and Language Teaching. Applied Linguistics Non Series.
Cambridge University Press, 2007. ISBN 9780521851466. URL https://books.google.it/books?id=O3k0tWLp5LYC.
W. Paley. Textarc: Showing word frequency and distribution in text. In Proceedings of IEEE Symposium on Information Visualization, Poster Compendium, IEEE CS Press, 2002.

A. S. Partington. Aims, Tools and Practices of Corpus Linguistics, pages
29. Handbooks of Linguistics and Communication Science 29.1. Centre
for the Study of Political Change (CIRCaP), Siena, Italy, 2006.
M. Phillips. Lexical Structure of Text. University of Birmingham, Birmingham, 1989.
R. Porst. Fragebogen: Ein Arbeitsbuch. Studienskripten zur Soziologie. VS
Verlag für Sozialwissenschaften, Wiesbaden, 2008. ISBN 9783531151786.
URL https://books.google.it/books?id=5i84wXjmmNIC.
R. Quirk. Towards a description of english usage. Transactions of the Philological Society, 59(1):4061, 1960.
P. Rayson, J. Mariani, B. Anderson-Cooper, A. Baron, D. Gullick, A. Moore,and S. Wattam. Towards interactive multidimensional visualisations
for corpus linguistics. Journal for Language Technology and Computational Linguistics, 31(1):2749, 2016. URL https://jlcl.org/content/
2-allissues/4-Heft1-2016/jlcl-2016-1-2RaysonEtAl.pdf.
J. O. Rüdiger. CorpusExplorer v2.0  Visualisierung prozessorientiert gestalten, pages 257268. Heidelberg University Publishing, 2018. doi: https://doi.org/10.17885/heiup.345.474.
266

A. Renouf. Corpus Development. Collins COBUILD. Collins ELT, 1987.
R. Reppen. Review of monoconc pro and wordsmith tools. Language Learning and Technology, 5(3):3236, 2001.
P. Riehmann, H. Gruendl, B. Froehlich, M. Potthast, M. Trenkmann, and
B. Stein. The netspeak wordgraph: Visualizing keywords in context. In
Proceedings of the 2011 IEEE Pacic Visualization Symposium, PACIFICVIS '11, pages 123130, Washington, DC, USA, 2011. IEEE Computer
Society. ISBN 978-1-61284-935-5. URL http://dl.acm.org/citation.
cfm?id=2015551.2015643.
J. O. Rüdiger.

Corpusexplorer.

corpusexplorer.de.

T. Russell-Rose and T. Tate.

Download, 2018.

URL http://

Designing the Search Experience:

Information Architecture of Discovery.

The

Morgan Kaufmann Publishers
Inc., San Francisco, CA, USA, 1st edition, 2013. ISBN 9780123969811,
9780123972880.

G. Sampson, editor. Corpus linguistics : readings in a widening discipline.
Continuum, London [u.a.], 2004.
D. Santos and A. Frankenberg-Garcia.
The corpus, its users andtheir needs: A user-oriented evaluation of COMPARA. International Journal of Corpus Linguistics, 12(3):335374, 2007. ISSN 13846655. URL http://www.ingentaconnect.com/content/jbp/ijcl/2007/
00000012/00000003/art00002.
R. Schäfer. Accurate and ecient general-purpose boilerplate detection for
crawled web corpora. Language Resources and Evaluation, 2016. doi:
10.1007/s10579-016-9359-2. URL http://link.springer.com/article/
10.1007/s10579-016-9359-2. online rst.
T. Schmidt and K. Wörner. Exmaralda  creating, analysing and sharing
spoken language corpora for pragmatic research. Pragmatics, 19(4):565
582, 2009.
N. Schwarz, H.-J. Hippler, B. Deutsch, and F. Strack. Response scales:Eects of category range on reported behavior and comparative judgments.
Public Opinion Quarterly, 49(3):388395, 1985. doi: 10.1086/268936. URL
http://poq.oxfordjournals.org/content/49/3/388.abstract.
M. Scott. WordSmith Tools. Oxford University Press, Oxford, UK, 1996.
M. Scott. Developing wordsmith. International Journal of English Studies,8(1):95106, Jun. 2008. URL https://revistas.um.es/ijes/article/
view/49111.
267

S. Sheehan. Visualising textual concordances with word mosaics andgraphs: Prototypes and evaluation methods.
Technical report,
School of Computer Science and Statistics, Trinity College Dublin, 08
2014. URL https://www.scss.tcd.ie/publications/tech-reports/
reports.14/TCD-CS-2014-05.pdf.
B. Shneiderman. The eyes have it: A task by data type taxonomy for information visualizations. In Proceedings of the 1996 IEEE Symposium
on Visual Languages, VL '96, pages 336, Washington, DC, USA, 1996.
IEEE Computer Society. ISBN 0-8186-7508-X. URL http://dl.acm.org/
citation.cfm?id=832277.834354.
J. Sinclair. Corpus, concordance, collocation. Describing English language.
Oxford University Press, 1991.
J. Sinclair. How to Use Corpora in Language Teaching. Studies in corpus
linguistics. John Benjamins Publishing, 2004. ISBN 9781588114907.
J.-P. Soehn, H. Zinsmeister, and G. Rehm. Requirements of a user-friendly,
general-purpose corpus query interface. In A. Witt, G. Rehm, T. Schmidt,
K. Choukri, and L. Burnard, editors, Proceedings of LREC 2008 Workshop 'Sustainability of Language Resources and Tools for Natural Language
Processing', pages 2732, May 2008.

E. St. John. A case for using a parallel corpus and concordancer for beginners
of a foreign language. Language Learning and Technology, 5(3):185203,
2001. URL https://dx.doi.org/10125/44573.
A. Stefanowitsch and S. T. Gries. Collostructions: Investigating the interaction between words and constructions. International Journal of Corpus
Linguistics, 8(2):209243, 2003.
D. Stone, C. Jarrett, M. Woodroe, and S. Minocha. User Interface Design and Evaluation. Interactive Technologies. Elsevier Science,
2005. ISBN 9780080520322. URL https://books.google.it/books?id=
VvSoyqPBPbMC.
F. Strack and L. Martin. Thinking, judging, and communicating: A process
account of context eects in attitude surveys., pages 123148. Springer,
New York, 1987.
R. Stuckardt. Towards best practice standards for enhanced knowledge discovery systems. In M. King and N. Underwood, editors, Proceedings of
LREC 2004 Workshop on User-oriented Evaluation of Knowledge discovery systems., pages 2428, 2004.

268

V. Suchomel and J. Pomikálek. Ecient web crawling for large text corpora.
In A. Kilgarri and S. Sharo, editors, Proceedings of the seventh Web
as Corpus Workshop (WAC7), pages 3943, Lyon, 2012. URL http://
sigwac.org.uk/raw-attachment/wiki/WAC7/wac7-proc.pdf.
J. Svartvik, editor. The London-Lund corpus of spoken English : description
and research. Lund studies in English 2. Lund University Press, Lund,
1990.
S. Tarp.

Lexicography in the Borderland Between Knowledge and Non-knowledge:

General Lexicographical Theory with Particular Focus on

Learner's Lexicography.

Lexicographica: Series maior. Niemeyer, TübinISBN 9783484391345. URL https://books.google.it/
books?id=YGXtzdtesxgC.
gen, 2008.

W. Teubert. My version of corpus linguistics. International Journal of Corpus Linguistics, 10(1):113, 2005.
V. Todorova and M. Chinkina. Slash a n-gram tendency viewer  visual exploration of n-gram frequencies in correspondence corpora. In
R. de Haan, editor, Proceedings of the ESSLLI 2014 Student Session, 26th
European Summer School in Logic, Language and Information, pages 229
239, Tübingen, Germany, 2014.
D. Todorovic. Gestalt principles. Scholarpedia, 3(12), 2008. URL 10.4249/
scholarpedia.5345.
E. Tognini-Bonelli. Corpus linguistics at work. John Benjamins Publishing
Company, 2001. ISBN 9781588110619.
C. Tribble. Teaching and language corpora: Quo vadis? In 10th Teachingand Language Corpora Conference (TALC), Warsaw, 2012.
E. Tufte. Envisioning Information. Graphics Press, Cheshire, CT, USA,1990. ISBN 0-9613921-1-8.
E. R. Tufte. Beautiful Evidence. Graphics Press, rst edition, May 2006.
ISBN 0-9613921-7-7.
N. L. Underwood. How to get the right systems to the right users. In M. Kingand N. Underwood, editors, Proceedings of LREC 2004 Workshop on Useroriented Evaluation of Knowledge discovery systems., pages 1821, 2004.
F. van Ham, M. Wattenberg, and F. Viegas. Mapping text with phrase
nets. IEEE Transactions on Visualization and Computer Graphics, 15(6):11691176, Nov.-Dec. 2009a.
269

F. van Ham, M. Wattenberg, and F. B. Viegas. Mapping text with phrase
nets. IEEE Transactions on Visualization and Computer Graphics, 15(6):11691176, Nov. 2009b. ISSN 1077-2626. doi: 10.1109/TVCG.2009.165.
URL https://doi.org/10.1109/TVCG.2009.165.
J. van Kuppevelt, U. Heid, and H. Kamp. Best practice in spoken language dialogue systems engineering: Introduction to the special issue.
Natural Language Engineering, 6(3&4):205212, 2000.
doi: 10.1017/S1351324901009391. URL http://dx.doi.org/10.1017/
S1351324901009391.
F. B. Viégas, S. Golder, and J. Donath. Visualizing email content: Portraying relationships from conversational histories. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems, CHI '06,
pages 979988, New York, NY, USA, 2006. ACM. ISBN 1-59593-3727. doi: 10.1145/1124772.1124919. URL http://doi.acm.org/10.1145/
1124772.1124919.
T. Virtanen. Corpora and discourse analysis, volume 2 of Handbooks of Linguistics and Communication Science 29.2, chapter 49, pages 10431069.
Mouton de Gruyter, Berlin, New York, 2008.
C. Ware. Information Visualization: Perception for Design. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2004. ISBN 1558608192.
M. Wattenberg and F. B. Viégas. The word tree, an interactive visual concordance. IEEE Transactions on Visualization and Computer Graphics, 14(6):12211228, Nov. 2008. ISSN 1077-2626. doi: 10.1109/TVCG.2008.172.
URL http://dx.doi.org/10.1109/TVCG.2008.172.
D. Wiechmann and S. Fuhs. Concordancing software. 2(1):107127, 06 2006.
T. Wilhelm, M. Burghardt, and C. Wol. "to see or not to see" - an interactive tool for the visualization and analysis of shakespeare plays. In
R. Franken-Wendelstorf, E. Lindinger, and J. Sieck, editors, Kultur undInformatik: Visual Worlds & Interactive Spaces, pages 175185. Verlag
Werner Hülsbusch, Glückstadt, 2013.
M. Wilkinson. Corpora, serendipity & advanced search techniques. The
Journal of Specialised Translation, 7:108122, 2007. URL http://www.
jostrans.org/issue07/art_wilkinson.php.
A. Wilson, P. Rayson, and T. McEnery. A Rainbow of Corpora: Corpus
Linguistics and the Languages of the World. Linguistics edition. LincomEuropa, 2002.
R. S. Witte and J. S. Witte. Statistics. Wiley, 11th edition edition, 2017.
270

M. Wynne, editor. Developing Linguistic Corpora: a Guide to Good
Practice.
Oxbow Books, Oxford, 2005. URL http://ahds.ac.uk/
linguistic-corpora/.
S. M. Yimam, I. Gurevych, R. Eckart de Castilho, and C. Biemann. Webanno: A exible,web-based and visually supported system for distributed
annotations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (System Demonstrations) (ACL 2013),
pages 16. Association for Computational Linguistics, Aug. 2013.
H. Yoon. An Investigation of StudentsÉxperiences with Corpus Technology
in Second Language Academic Writing. unpublished PhD thesis, The Ohio
State University, Columbus, OH, 2005. URL https://books.google.it/
books?id=7p3ijwEACAAJ.
J. Zhao, F. Chevalier, C. Collins, and R. Balakrishnan. Facilitating discourse
analysis with interactive visualization. IEEE Trans. on Visualization andComputer Graphics (Proc. of the IEEE Conf. on Information Visualization(InfoVis), 18(12):26392648, Dec. 2012. doi: 10.1109/TVCG.2012.226.

C. Zinn. The clarin language resource switchboard. In CLARIN 2016 Annual
Conference, Aix-en-Provence, France, 2016.
F. Zipser and L. Romary. A model oriented approach to the mapping ofannotation formats using standards. In Proc. LREC-2010 Workshop on
Language Resource and Language Technology Standards (LR&LTS 2010),
Valetta, Malta, May 2010.

271

272

-

Appendix

A-

Interview data

273

274

Table A.1: Guiding questions for interviews 1 (I1)

1
2
3
5
_a
_b
_c
_d
_e
_f
_g
_h
6
7
8a
_b
9
10
11
12

Research background of the user
What was the reason for you to start working with corpora?
When did you start working with corpora?
Beforehand, did you do any linguistic studies that were not based on
corpora?
Analysis procedure of the corpus study
How did you go about the corpus analysis? What analysis steps did you
carry out and how?
Which tools did you employ for which step?
How did you process and continue with the results of each analysis step?
Did you analyze results within a tool or based on a data export?
How did you pick the examples for closer analysis?
What information did you take into account for classifying results?
How did you store classications and which information did you include?
Did you ever reimport manually processed results into a query tool?
...
Usage of corpus tools and related requirements
What types of corpus querying tools did you use so far?
What did you like or dislike about the tools?
Did you have particular requirements or ideas what was missing in thetools?
Do you have an example in mind where you felt that you need to prepare
your own scripts to achieve some research insight?
What would you say in general terms, is the most important type of
support that a tool should provide for your type of corpus studies?
How did you organize transitions between tools?
How do you generally document your workows?
Further comments on corpus methodology
Do you have any other further general ideas, comments or opinions about
concordancing or corpus query tools, and on how researchrers approach
corpus research?

275

Table A.2: Guiding questions for interviews 2 (I2)
Research background of the user−→ investigated in pre-questionnaire, see Table A.3

1
3

2
4
5
6
7
8
9
10
11
12
13

Research topic
What is your main research topic that you are approaching with the helpof corpora?
Corpus information types
What type of corpus information are you interested in? How do the
aspects of your interest occur in the corpus? (e.g. information of interest:
context of a word, to nd regularities in context; occurrence in corpus:
search word in running text, e.g. concordance)
Analysis goals, tasks, and actions
What are your concrete goals, when approaching a corpus? What are
single working steps and what are you aiming to achieve with them?
You indicated that you are primarily interested in qualitative/quantitative analyses. What are typical qualitative/quantitative analysis steps
you are carrying out?
Do you do exploratory analyses? explanation: corpus searches without
concrete hypothesis/expected result/aim
How are you doing exploratory searches? Could you give me examples
of exploratory searches?
What is your motivation for doing exploratory searches? What are yourexpectations related to it?
Theoretical assumptions and methodology
What do you consider the potential and limits of corpus-based linguistic
analyses?
In general terms, what are factors concerning corpus work that facilitateor hinder to reach your research goals?
What is supportive or creates diculties in corpus analyses, i.e to accessrelevant information and reach an interpretation?
According to you, what are necessary conditions for a corpus analysis
study to be successful? What needs to hold so that you would consider
the analysis a success?
Can you name methodological principles that you are following?
Do you consider results and insight as reliable and generalizable?

276

Table A.3: Written pre-questionnaire for interviews 2 (I2)

1
2
3
4
5
6
7

8

9

10

11
12
13

Research background of the user
Name and research institution
Which degree do you hold and what is your professional background?
What is your main research topic?
For how many years have you been working with corpora?
In your current research, what share or your research work is based on
corpora (percentage of your research studies)?
How often do you make use of corpora for your research?
_several times a week; _at least once a week; _at least once a month;_several times a year;
Do you mainly carry out your corpus research as a team eort or alone?
In case what is your role within the research team? (e.g. denition of
the research question, planning of the corpus investigation, quering the
corpus, programming of tools, analysis of search results, etc.)
How skilled are you with using computers? (as user not as programmer)_very skilled; _adequately skilled; _adequately skilled, but with occasional diculties; _I often need of support by somebody; _I hardly workwith computers;
Please indicate, which of the following tools you have yet made use offor analyzing corpora:_Query tools with graphical user interface (e.g. WordSmith, AntConc,
online search interfaces); _Query tools with command line access (e.g.
CQP der CorpusWorkbench, DDC, . . . ); _specialized programs or
scripts which are called on the command line; _tools programmed bymyself; _tools programmed by my colleagues;
How skilled are you in using corpus query tools?
_very skilled; _adequately skilled; _adequately skilled, but with occasional diculties; _I often need of support by somebody; _I hardly workwith corpus query tools;
Are you mainly intersted in quantitative or qualitative aspects of corpus
data?
Which annotations are usually available in the corpora that you typicallywork with?
Are you familiar with statistical measures?
_I did not yet look into this; _I am capable to understand and interpret
results of statistical analyses; _I am capable to choose and apply relevant
statistical measures;

277

Table A.4: Formal characteristics of interviewees (I1 and I2); IDs 1.01-12
refer to the rst interview series, and IDs 2.01-08 to the second series.

ID
1.01
1.02
1.03
1.04
1.05
1.06
1.07
1.08
1.09
1.10
1.11
1.12
2.01
2.02
2.03
2.04
2.05
2.06

2.07
2.08

Professional
background
English literature, learning and teaching in higher
education
linguistics and English
language, corpus linguistics, computer sciencelinguistics, with minor informatics
applied linguistics, translation studies
English and applied linguistics, translation studies
general linguistics, German linguistics, minor incomputational linguistics
linguistics
linguistics
linguistics
English philology
linguistics
English literature
Slavic studies, linguistics,
minor in computer scienceand psychology
English linguistics
High school teaching ofEnglish and French
English linguistics
English studies, language
learning, linguistics
linguistics, psychology,
computer science, computational
linguistics,
English linguistics
linguistics
linguistics and computerscience

CorpusProgramming/
experience scripting skills
3 years
no skills

Status of CL
as methodside method> 6 years

skilledmain method

5 years

skilledmain method

8 years

limited skills

main method> 6 years

no skills

main method

3 years

limited skills

main method> 6 years

10 years
2 years
8 years
4 years
3 years
10 yearsno skills
limited skills
no skills
no skills
no skills
no skills
limited skills

main method
main methodside method
main method
main methodside method
main method

20 years
12 yearsno skills
skilledmain method
main method

15 years
10 years

limited skills
skilledmain method
main method

15 years

limited skills

side method

11 years
13 years

skilledskilledmain method
main method

278

-

Appendix

B-

Questionnaire data

279

280

Table B.1: Questionnaire structure
Nb Question
USER

Answer type

Q7
Q8
Q9
Q10

single select, 3 options
single select, Boolean yes/notext eld
text eld

Q2
Q3
Q4
Q5
Q6

What is your education/professional background?
What is your current professional activity?
For how many years have you regularly been working with corpora?
How often do you access corpora?
How does the corpus-based approach relate to other methods you are applying in your research(i.e. experimental methods, e.g. speaker judgements; theoretical work; etc.)?
How do you feel about using computers and trying out new tools/software?
Do you have (basic) programming skills?
Which (human) language(s) do you work on?
Please indicate your year of birth.

multiple select, 8 options
text eld
single select, 4 options
single select, 3 options
single select, 3 options

When doing corpus analyses, how often do you make use of the following tools? (7 answer
categories)
How important are for you the following features and characteristics of a tool for corpus analysis?
(10 answer categories)

5-point rating, 'very often  never'

Q13

What are your motivations for doing corpus-based analyses? (4 answer categories)

Q14
Q15

According to your experiences and perspectives what are crucial conditions for a valid andsuccessful corpus analysis?
How often do you carry out linguistic analyses based on corpora?

5-point rating, 'very much  not atall' + 'other'text eld

Q16
Q17

Dening a linguistic hypothesis - How often do you carry out this task?
Building corpora from scratch - How often do you carry out this task?

TOOLS
Q11
Q12

METHODOLOGY

TASKS

5-point rating, 'very important  notimportant' + 'other'

5-point rating, 'very often  never'
5-point rating, 'very often  never'
5-point rating, 'very often  never'Continued on next page

Table B.1  continued from previous pageNb Question
Q18
Q19
Q20

Getting to know the corpus to work with - How often do you carry out this task?
Exploring a corpus - How often do you carry out this task?
What are your aims in exploratory corpus analyses? (5 answer categories)

Q21

When 'Exploring a corpus', how often do you carry out the following actions? (6 answer
categories)
Organizing and manually processing/manipulating extracted examples - How often do you carryout this task?
When 'Organizing and manually processing/manipulating extracted examples', how often do
you carry out the following actions? (9 answer categories)
Inspecting the extracted data - How often do you carry out this task?
When 'Inspecting the extracted data', how often do you carry out the following actions? (6answer categories)
Quantitative analysis of extracted data - How often do you carry out this task?
When doing 'Quantitative analysis of extracted data', how often do you carry out the following
actions? (6 answer categories)
Documenting and reporting ndings of a corpus study - How often do you carry out this task?
When 'Documenting and reporting ndings of a corpus study', how often do you carry out thefollowing actions? (6 answer categories)
Are there other primary tasks that are relevant to your corpus work?
Given that you dont't do linguistic analyses, what are primary tasks that you carry out in yourcorpus work?

Q22
Q23
Q24
Q25
Q26
Q27
Q28
Q29
Q30
Q31

TYPES OF DATA
Q32

How relevant are the following data formats for your corpus analyses? (8 answer categories)

Answer type5-point rating, 'very often  never'
5-point rating, 'very often  never'
5-point rating, 'very much  not atall' + 'other'
5-point rating, 'very often  never'+ 'other'5-point rating, 'very often  never'
5-point rating, 'very often  never'+ 'other'5-point rating, 'very often  never'
5-point rating, 'very often  never'+ 'other'5-point rating, 'very often  never'
5-point rating, 'very often  never'+ 'other'5-point rating, 'very often  never'
5-point rating, 'very often  never'+ 'other'text eld
text eld
5-point rating, 'very relevant  irrelevant'; 'availability' checkbox +'other'Continued on next page

Table B.1  continued from previous pageNb Question
Q33

How relevant are the following annotations for you for analyzing data retrieved from a corpus?
(18 answer categories)

Q34
Q35
Q36

How often is it relevant to your corpus analysis work to look at dierent data formats in parallel?
More specically how relevant is the parallel display of the following combinations of corpus
data? (10 answer categories)
Do you have any other comments concerning relevant data types in corpus linguistic analysis?

Q37

How do you assess the potential use of visualizations for supporting corpus linguistic analysis?

Q38

What are aspects of the corpus analysis process that you can envision to benet most from(interactive) visual forms of data presentation?
How useful do you consider visualizations for the display of and interaction with the following
corpus data types? (8 answer categories)
How useful do you consider visualizations in supporting you in the following corpus analysis
tasks? (8 answer categories)

VISUALIZATION

Q39
Q40

Answer type5-point rating, 'very relevant  irrelevant'; 'availability' checkbox +'other'5-point rating, 'very often  never'
5-point rating, 'very relevant  irrelevant' + 'other'text eld
5-point rating, 'very useful  not at
all useful'text eld
5-point rating, 'very useful  not at
all useful' + 'other'
5-point rating, 'very useful  not at
all useful' + 'other'Figure B.1: Questionnaire survey, page 1

Figure B.2: Questionnaire survey, page 2

Figure B.3: Questionnaire survey, page 3

Figure B.4: Questionnaire survey, page 4

Figure B.5: Questionnaire survey, page 5

Figure B.6: Questionnaire survey, page 6

Figure B.7: Questionnaire survey, page 7

Figure B.8: Questionnaire survey, page 8

Figure B.9: Questionnaire survey, page 9

Figure B.10: Questionnaire survey, page 10

Figure B.11: Questionnaire survey, page 11

Figure B.12: Questionnaire survey, page 12

Figure B.13: Questionnaire survey, page 13

Figure B.14: Questionnaire survey, page 14

Figure B.15: Questionnaire survey, page 15

Figure B.16: Questionnaire survey, page 16

Figure B.17: Questionnaire survey, page 17

Figure B.18: Questionnaire survey, page 18

Figure B.19: Questionnaire survey, page 19

Figure B.20: Questionnaire survey, page 20

Figure B.21: Questionnaire survey, page 21

Table B.2: Questions related to the USERS
Nb
Q2

Q3
Q4

Q5

Q6

Q7

Q8
Q9
Q10

Question and answer options

What is your education/professional background?
- linguistics- literary studies- language teaching- translation/interpreting- lexicography/terminology- computational linguistics- computer science/mathematics- other ()
What is your current professional activity?
-()
For how many years have you regularly beenworking with corpora?
- less than 1 year
- 1-3 years- 4-8 years- more than 8 years
How often do you access corpora?
- several times a week
- several times a month- several times a year
How does the corpus-based approach relate toother methods you are applying in your research?
Corpora are ...
- the main methodology in my research activities- one research methodology employed besides
other approaches- a methodology employed marginally and lessthan other approaches
How do you feel about using computers and trying out new tools/software?
- I enjoy it- It is OK for me- I sometimes feel intimidated by computers
Do you have (basic) programming skills?
- yes
- no
Which (human) language(s) do you work on?
-()
Please indicate your year of birth.
-()

please specify
please specify

please specify
please specify

Answer typee
multiple selecttext eld
single selectsingle selectsingle selectsingle selectsingle selecttext eld
text eldTable B.3: Questions related to the TOOLSTOOLSNb Question
Q11 When doing corpus analyses, howoften do you make use of the following tools?

Q12

Answer options
web interfaces to online corpora, stand-alone
concordancing software, locally installed corpus
query infrastructures, custom scripts for the extraction of data, custom scripts for quantitative
analyses, generic oce programs and text processing software, statistical software
How important are for you the export functionalities, reimport functionalities,
following features and character- exible query language, speed and scalability,
iscs of a tool for corpus analysis? support for a formalized and replicable analysis
procedure, statistics, initiating new awarenesses,
possibility to generate statistics on the data,
data retrieval, support for publication-quality
versions of visualizations of data/examples, easy
to learn and use
other (please specify )Answer type5-point Likert
scale, [1:'very
often'
to5:'never']5-point Likert
scale, [1:'very
important' to
5:'not important']

text eldTable B.4: Questions related to the METHODOLOGYMETHODOLOGY
Nb Question
Answer options
Q13 What are your motivations for getting ideas/inspiration from empirical data,
doing corpus-based analyses?
getting quantitative information for a linguistic
phenomenon, gathering qualitative evidence anddetails of actual language use, gather data to be
integrated into an NLP tool
other aims for using corpora (please specify )Q14 According to your experiences (please specify )and perspectives what are crucial
conditions for a valid and successful corpus analysis?
Q20 What are your aims in ex- get ideas/inspiration for new research topics,
ploratory corpus analyses?
take a data-driven approach to detect patterns, etc., get an understanding of the corpus,
get a rst understanding of a linguistic phenomenon/topic, go into detail on a linguistic
phenomenon/topic
other (please specify )Answer type5-point Likert
scale, [1:'very
much' to 5:'notat all']text eld
text eld

5-point Likert
scale, [1:'very
much' to 5:'notat all']text eldTable B.5: Questions related to the TASKSTASKS IN CORPUS ANALYSIS STUDIES
Nb Question
Answer options
Q16 Dening a linguistic hypothesis
How often do you carryout this task?
Q17 Building corpora from scratch
How often do you carryout this task?
Q18 Getting to know the corpus to work How often do you carrywithout this task?
Q19 Exploring a corpus
How often do you carryout this task?
Q22 Organizing and manually processing/- How often do you carrymanipulating extracted examples
out this task?
Q24 Inspecting the extracted data
How often do you carryout this task?
Q26 Quantitative analysis of extracted data How often do you carryout this task?
Q28 Documenting and reporting ndings of How often do you carrya corpus study
out this task?

Answer type5-point Likert scale, [1:'very
often' to 5:'never']5-point Likert scale, [1:'very
often' to 5:'never']5-point Likert scale, [1:'very
often' to 5:'never']5-point Likert scale, [1:'very
often' to 5:'never']5-point Likert scale, [1:'very
often' to 5:'never']5-point Likert scale, [1:'very
often' to 5:'never']5-point Likert scale, [1:'very
often' to 5:'never']5-point Likert scale, [1:'very
often' to 5:'never']Table B.6: Questions on the ACTIVITIES related to TASKSACTIVITIES FOR TASKS IN CORPUS ANALYSIS STUDIES
Nb Question
Answer options
Answer type
Q21 When 'Exploring a corpus', how inspecting samples of source documents, ex- 5-point Likert
often do you carry out the follow- ploratory corpus querying and successive query scale, [1:'very
ing actions?
renement, eyeballing sets of query results, in- often'tospect details, do quantitative analyses of search 5:'never']results, systematically extract statistics for specied linguistic parameters
other (please specify )text eld
Q23 When 'Organizing and manu- randomizing results/obtaining a random subset, 5-point Likert
ally processing/manipulating ex- subsequent querying on results set, sorting by scale, [1:'very
tracted examples', how often do words of the left/right context, sorting by anno- often'
toyou carry out the following ac- tations of the left/right context, grouping results 5:'never']tions?
by metadata, manually classifying search results
according to a set of categories, manually sorting
results into groups, deleting irrelevant hits fromsearch results, correcting automatic annotations
other (please specify )text eld
Continued on next page

Table B.6  continued from previous pageNb Question
Answer options
Q25 When 'Inspecting the extracted inspecting results one-by-one, inspecting (subdata', how often do you carry out )sets of results, comparing (sub-)sets of results,
the following actions?
highlighting parts of the results, going back towider context or source documents of the corpus,
referring to additional information derived from
corporaother (please specify )Q27 When doing 'Quantitative analy- calculate and compare absolute and relasis of extracted data', how often tive frequencies, calculate word associations/codo you carry out the following ac- occurrences, calculate 'keyness' of words/linguistions?
tic structures, statistical hypothesis testing, calculate the distribution and dispersion of words,
create graphics of statistical data for analysis
other (please specify )Q29 When 'Documenting and report- describing the analysis procedure/methodology,
ing ndings of a corpus study', describing the data under analysis, explaining
how often do you carry out the ndings of the study, selecting examples to ilfollowing actions?
lustrate the ndings, presenting selected examples for illustration of the ndings, discussing thegeneralizability of results
other (please specify )Answer type5-point Likert
scale, [1:'very
often'
to5:'never']text eld
5-point Likert
scale, [1:'very
often'
to5:'never']text eld
5-point Likert
scale, [1:'very
often'
to5:'never']text eldTable B.7: Questions related to the DATATYPES OF DATA
Nb Question
Q32 How relevant are the following
data formats for your corpus
analyses?

Answer options
Answer typesamples of documents that the corpus is com- 5-point Likert
posed of for manual inspection, full set of docu- scale, [1:'very
ments that the corpus is composed of for process- relevant'toing by custom scripts, KWIC/concordances with 5:'irrelevant']
wide context, KWIC/concordances with narrow and check box
context, lists of words/complex search strings 'yes, usuallyand their frequencies, lists of lemmas/parts-of- available'
speech and their frequencies, lists of n-grams andtheir frequencies, lists of collocations and theirfrequencies
other relevant data formats (please specify )text eld
Continued on next page

Table B.7  continued from previous pageNb Question
Answer options
Q33 How relevant are the following lemma, part-of-speech, advanced morphologicalannotations for you for analyzing information, syntactic information - dependency
data retrieved from a corpus?
structures, syntactic information - phrase structures/constituents (treebanks), syntactic information - chunk parses, semantic information word sense disambiguation/lexical semantics, semantic information - role labeling/semantic tagging, discourse information - coherence markers,
discourse information - coreference resolution,
pragmatic information, named entity tags, custom annotations for specic aspects of the data,
metadata on the document level, metadata on
the sentence level, sentence boundaries, section
boundaries, information on corpus design/corpus composition/annotation guidelines
other relevant annotations (please specify )Q34 How often is it relevant to yourcorpus analysis work to look atdierent data formats in parallel?

Answer type5-point Likert
scale, [1:'very
relevant'
to5:'irrelevant']
and check box'yes, usuallyavailable'

text eld
5-point Likert
scale, [1:'very
often'
to5:'never']Continued on next page

Table B.7  continued from previous pageNb Question
Answer options
Q35 More specically, how relevant is source text document and metadata, source text
the parallel display of the follow- document and linguistic annotations, KWIC reing combinations of corpus data? sults and metadata, KWIC results and linguistic annotations, KWIC results and source text
documents, KWIC results and distribution ofhits over corpus/document, KWIC results andfrequency lists of words/collcations, frequency
lists of word/complex linguistic items and related KWIC lines or corpus source documents,
frequency lists of words/complex linguistic items
and their distribution over the corpus/documents
other combinations (please specify )Q36 Do you have any other comments (please specify )concerning relevant data types in
corpus linguistic analysis?

Answer type5-point Likert
scale, [1:'very
relevant'
to5:'irrelevant']

text eld
text eldTable B.8: Questions related to VISUALIZATIONSVISUALIZATION OF CORPUS DATA
Nb Question
Answer options
Q37 How do you assess the potential
use of visualizations for supporting corpus linguistic analysis?
Q38

Q39

What are aspects of the corpus
analysis process that you can envision to benet most from (interactive) visual forms of data
presentation?
How useful do you consider visualizations for the display and interaction with the following corpus data types?

(You can give reference to analysis aims, tasks
and involved data types/data formats,specify )Answer type5-point Likert
scale, [1:'very
useful'to
5:'not at all
useful']text eldpleasedocumentation on corpus composition and anno- 5-point Likert
tation layers, single source documents, KWIC/- scale, [1:'very
concordances with or without linguistics anno- useful'totations, lists of words n-grams collocations etc., 5:'not at all
statistics on linguistic features, statistics com- useful']paring linguistic features, distribution of linguistic items over the corpus, dispersion of linguistic
items in documents
other data types for visualization (please specify ) text eldContinued on next page

Table B.8  continued from previous pageNb Question
Answer options
Q40 How useful do you consider vi- hypothesis building, getting to know a corpus,
sualizations in supporting you exploring a corpus, organizing and manual proin the following corpus analysis cessing/ manipulating extracted examples, intasks?
specting data extracted from corpora, quantitative analysis of data extracted from corpora,drawing conclusions from analyzed data, documenting and reporting ndings of a corpus study
other corpus analysis tasks to be supported byvisualization (please specify )Answer type5-point Likert
scale, [1:'very
useful'to
5:'not at all
useful']text eldTable B.9: Descriptive statistics for all Likert items of the survey
quest

Q11_1
Q11_2
Q11_3
Q11_4
Q11_5
Q11_6
Q11_7
Q12_1
Q12_2
Q12_3
Q12_4
Q12_5
Q12_6
Q12_7
Q12_8
Q12_9
Q12_10Q12_other
Q13_1
Q13_2
Q13_3
Q13_4
Q13_other
Q14
Q15
Q16
Q17
Q18
Q19
Q20_1
Q20_2
Q20_3
Q20_4
Q20_5
Q20_other
Q21_1
Q21_2
Q21_3
Q21_4
Q21_5
Q21_6
Q21_other
Q22
Q23_1
Q23_2

median
3
4
4
3
3
2
3
1
2
1
2
2
2
3
2
2
2

IQR
3
3
3
4
4
2
2.25
1
2
1
1
2
2
1
1
1.5
2

mean
2.69
3.17
3.5
3.09
3.19
2.19
2.78
1.6
2.28
1.75
1.89
2.25
2.04
2.81
2.39
2.39
2.08

std
1.43
1.53
1.57
1.72
1.7
1.39
1.35
1.06
1.3
0.99
0.85
1.16
1.03
1.07
1.11
1.22
1.13

NA
0
0
0
0
2
0
0
1
1
1
1
1
1
4
3
1
1

1
1
1
2.5

1
1
1
3

1.71
1.46
1.89
2.75

0.96
0.85
1.15
1.51

0
0
1
2

1
2
2
2
2
2
2
2
2
2

1
2
2.25
2
2
1
1
1
2
2

1.66
2.28
2.41
2.31
2.43
2.48
1.78
2.55
2.04
1.97

0.86
1.16
1.28
1.01
1.22
1.16
0.96
1.09
0.99
1.05

1
0
0
0
0
7
7
7
7
7

2
2
2
2
2
2

2
1
2
2
1
2

2.18
1.92
2.06
2.11
1.89
2.38

1.12
1.04
1.18
1.02
0.97
1.19

7
7
7
7
7
7

2
3
2

2
2
1

2.06
2.94
2.5

count

10

12
72

2

1
1.2
0
1.37
5
1.2
5
Continued on next page

Table B.9  continued from previous pagequest
median IQR
mean
std
Q23_3
Q23_4
Q23_5
Q23_6
Q23_7
Q23_8
Q23_9
Q23_other
Q24
Q25_1
Q25_2
Q25_3
Q25_4
Q25_5
Q25_6
Q25_other
Q26
Q27_1
Q27_2
Q27_3
Q27_4
Q27_5
Q27_6
Q27_other
Q28
Q29_1
Q29_2
Q29_3
Q29_4
Q29_5
Q29_6
Q29_other
Q30
Q31

quest

Q32_1_1
Q32_2_1
Q32_3_1
Q32_4_1
Q32_5_1
Q32_6_1
Q32_7_1
Q32_8_1
Q32_other

2
3
2
2
2
2
4

2.25
2
3
2
2
2
2.5

2.47
2.84
2.44
2.07
2.22
2.25
3.28

1.36
1.31
1.33
1.26
1.35
1.33
1.4

NA
4
4
6
6
5
5
5

1
2
2
2
2
2
2

1
1
2
2
2
2
2

1.69
1.85
2.09
2.11
2.45
2.21
2.59

0.99
1.04
1.06
1.21
1.37
1.13
1.3

0
3
4
4
3
3
4

2
1
2
3
2
3
2

2
1
2
2
2.75
2
3

2.01
1.66
2.46
2.93
2.53
2.9
2.36

1.18
1.04
1.33
1.4
1.38
1.38
1.4

1
6
6
6
6
5
5

2
1
1
1
1
1
2

2
1
1
1
1
1
1

2
1.49
1.52
1.52
1.46
1.46
1.77

1.02
0.81
0.84
0.93
0.8
0.83
0.98

2
3
3
3
3
3
3

median
2
2
2
2
2
2
2
2

IQR
2
2
2
2
1.25
2
2.75
2

mean
2.3
2.31
2.3
2.43
1.96
1.98
2.61
2.37

std
1.33
1.38
1.44
1.38
1.15
1.11
1.35
1.34

NA
4
5
4
5
4
6
6
4

count

2

1

3

2
15
0

avail.

count
7
Continued on next page

Table B.9  continued from previous pagequest
median IQR
mean
std
quest
median IQR
mean
std
Q33_1_1
Q33_2_1
Q33_3_1
Q33_4_1
Q33_5_1
Q33_6_1
Q33_7_1
Q33_8_1
Q33_9_1
Q33_10_1
Q33_11_1
Q33_12_1
Q33_13_1
Q33_14_1
Q33_15_1
Q33_16_1
Q33_17_1
Q33_18_1

Q33_other
Q34
Q35_1
Q35_2
Q35_3
Q35_4
Q35_5
Q35_6
Q35_7
Q35_8
Q35_9
Q35_10Q35_other
Q36
Q37
Q38
Q39_1
Q39_2
Q39_3
Q39_4
Q39_5
Q39_6
Q39_7
Q39_8
Q39_other
Q40_1

1
1
3
3
3
3
3
3
4
3
4
3
3
2
2
1
3
1

1
1
3
2.25
2
2
3
2
2
2
2
2
2
2
3
1
2.75
1

1.63
1.62
2.56
2.74
2.94
3.1
2.84
3.01
3.33
3.29
3.52
3.04
2.9
2.05
2.55
1.94
2.73
1.79

1
0.95
1.32
1.34
1.42
1.35
1.43
1.48
1.36
1.38
1.34
1.44
1.47
1.28
1.45
1.29
1.39
1.2

NA
NA
6
7
4
4
4
7
3
7
6
6
6
8
5
3
6
5
6
4

3
3
2
3
2
3
3
3
2
2
2

2
3
2
3
2
2
2
2
2
3
2

2.84
2.58
2.21
2.62
2.32
2.6
2.58
2.5
2.63
2.58
2.45

1.33
1.37
1.26
1.3
1.21
1.31
1.24
1.26
1.34
1.36
1.27

1
18
17
18
18
18
19
18
17
18
20

1

1

1.68

0.93

0

2
3
2
2
1
1
1
2

2
2
2
2
1
1
1
2

2.37
2.79
2.33
2.1
1.59
1.53
1.62
2.03

1.27
1.19
1.23
1.14
0.86
0.83
0.87
1.09

8
7
8
5
5
6
7
7

2

2

2

count

avail.

count
8

5
5
40

3
1.12
7
Continued on next page

Table B.9  continued from previous pagequest
median IQR
mean
std
Q40_2
Q40_3
Q40_4
Q40_5
Q40_6
Q40_7
Q40_8
Q40_other

2
1
2
2
1
2
1

2
2
3
1
1
1
1

1.93
1.89
2.52
1.96
1.72
1.86
1.61

1.09
1.13
1.34
1.01
1
1.04
0.89

NA
6
5
7
7
6
7
6

count

5

-

AppendixC-

Visualization examples

321

Figure C.1:

GistIcons visualization,

as presented in DeCamp et al.

((2015)), p. 1

Figure C.2: FacetAtlas, as presented in Cao et al. ((2010)), Fig. 1, p.
1172

Figure C.3: TextArc visualization, as dened in Paley ((2002)), Fig. 1

Figure C.4: InfoSky visualization, as dened in Granitzer et al. ((2004)),
Fig. 1, p. 127

Figure C.5: xHIPP visualization, as dened in Dias and Minghim ((2018)),
Fig. 4, p. 366

Figure C.6: Jigsaw visualization, as dened in Görg et al. ((2013)), Fig.
2, p. 1652

Figure C.7: MiTextExplorer, as dened in ((O'Connor, 2014)), Fig. 1

Figure C.8: FeatureLens visualization, as in ((Don et al., 2007)), Fig. 2,
p. 215Figure C.9: Themeriver visualization, as dened in ((Havre et al., 2000)),
Fig. 4, p. 120

Figure C.10: Parallel Tag Clouds, as dened in ((Collins et al., 2009b)),Fig. 1, p. 92

Figure C.11: DAViewer visualization, as dened in ((Zhao et al., 2012)),Fig. 1, p. 2639

Figure C.12: Themail visualization, as dened in ((Viégas et al., 2006)),
Fig. 3, p. 983

Figure C.13: Visualization of character and discourse time in 'Hamlet', asdened in ((Wilhelm et al., 2013)), Fig. 3, p. 181

Figure C.14:

Discourse recurrence plot, as dened in ((Angus et al.,2012)), Fig. 4, p. 992

Figure C.15:

Splitstree visualization to display syntagmatic xedness ofmulti word expressions, as dened in ((Herold and Stathi, 2007)), Fig. 5, p.
13

Figure C.16: Syntactic dependency structure visualization, as dened in((Culy et al., 2011)), Fig. 4, p. 167

Figure C.17: Double Tree visualization, as dened in ((Culy and Lyding,2010)), Fig. 3, p. 100

Figure C.18: Netspeak Wordgraph visualization, as dened in ((Riehmann
et al., 2011)), Fig. 7, p. 127

Figure C.19:

Parallel Coordinates visualization, taken from https: //visflow. org/ node/ visualization/ parallel-coordinates. html

FigureC.20:

FigureC.21:

TreeMap

visualization

taken

from

// www. grapecity. com/ wijmo/ docs/ Topics/ Chart/ Advanced/
SpecialCharts/ TreeMap

Sunburst

visualization

taken

from

dataforvisualization. com/ charts/ sunburst-diagram/

https:

https: //-AppendixD-

Sample data and 'R' scripts

333

# Example 'R' s c r i p t for the calculation of mean,# median , inter q u a r t i l e range , standard deviation# and empty values for question 'Q11_1 'q u e s t d a t a = read . csv ( " n u m e r i c a l =Qnb . t x t " )q u e s t <= c ( ' Q11_1 ' )mean <= c ( round ( mean ( q u e s t d a t a $Q11_1 , na . rm=TRUE) , d i g i t s =2))
median <= c ( median ( q u e s t d a t a $Q11_1 , na . rm=TRUE) )IQR <= c (IQR( q u e s t d a t a $Q11_1 , na . rm = TRUE) )s t d <= c ( round ( sd ( q u e s t d a t a $Q11_1 , na . rm=TRUE) , d i g i t s =2))na <= c ( sum ( i s . na ( q u e s t d a t a $Q11_1 ) ) )a l l . data <= data . frame ( quest , mean , median , IQR, std , na )write . csv ( a l l . data , " out =n u m e r i c a l =Qnb . t x t " )# Example 'R' s c r i p t for comparing mean values of# Q32 and Q33 in relation to the indication# of ' a v a i l a b i l i t y 'mean_a l l <=c ( round ( mean ( q u e s t d a t a $Q32_1_1 , na . rm=TRUE) , d i g i t s =2))
mean_a v a i l <=c ( round ( mean ( q u e s t d a t a [ which ( q u e s t d a t a $Q32_1_2 == " 1 " ) ,names ( q u e s t d a t a ) %i n% c ( "Q32_1_1 " ) ] , na . rm=TRUE) , d i g i t s =2))
mean_n o t a v a i l <=c ( round ( mean ( q u e s t d a t a [ which ( q u e s t d a t a $Q32_1_2 == " 0 " ) ,names ( q u e s t d a t a ) %i n% c ( "Q32_1_1 " ) ] , na . rm=TRUE) , d i g i t s =2))
a l l . means <= data . frame ( mean_a l l , mean_a v a i l , mean_n o t a v a i l )write . csv ( a l l . means , " out =means= a v a i l = n o t a v a i l . t x t " )The le 'numerical_Qnb.txt' is a comma separated le which contains thequestionnaire results to all Likert scale questions, preceded by information
on user-related questions. The rst line contains the question numbers, while
each of the following lines contains the values for one respondent.

# Example 'R' s c r i p t for the calculation of a bar plot
# for question 'Q11_1 'library ( psych )library ( l i k e r t )Input =("
web_i n t e r f a c e s s t a n d a l o n e_c o n c o r d a n c i n g
l o c a l_query_i n f a s t r u c t u r e s c r i p t s_f o r_data_e x t r a c t i o n
s c r i p t s_f o r_q u a n t i f i c a t i o n o f f i c e _programs s t a t i s t i c a l _s o f t w a r e
4 5 1 2 3 5 5
1 5 5 1 1 5 5
5 4 4 1 1 4 3")
Data = read . table ( t e x t C o n n e c t i o n ( Input ) , h e a d e r=TRUE)
Data $ web_i n t e r f a c e s =factor ( Data $web_i n t e r f a c e s ,l e v e l s = c ( "1 " , " 2 " , " 3 " , " 4 " , " 5 " ) ,ordered = TRUE)
Data $ s t a n d a l o n e_c o n c o r d a n c i n g =factor ( Data $ s t a n d a l o n e_c o n c o r d a n c i n g ,l e v e l s = c ( "1 " , " 2 " , " 3 " , " 4 " , " 5 " ) ,ordered = TRUE)
Data $ l o c a l_query_i n f a s t r u c t u r e =factor ( Data $ l o c a l_query_i n f a s t r u c t u r e ,l e v e l s = c ( "1 " , " 2 " , " 3 " , " 4 " , " 5 " ) ,ordered = TRUE)
Data $ s c r i p t s_for_data_e x t r a c t i o n =factor ( Data $ s c r i p t s_for_data_e x t r a c t i o n ,l e v e l s = c ( "1 " , " 2 " , " 3 " , " 4 " , " 5 " ) ,ordered = TRUE)
Data $ s c r i p t s_for_q u a n t i f i c a t i o n =factor ( Data $ s c r i p t s_for_q u a n t i f i c a t i o n ,l e v e l s = c ( "1 " , " 2 " , " 3 " , " 4 " , " 5 " ) ,ordered = TRUE)
Data $ o f f i c e _programs =factor ( Data $ o f f i c e _programs ,l e v e l s = c ( "1 " , " 2 " , " 3 " , " 4 " , " 5 " ) ,ordered = TRUE)
Data $ s t a t i s t i c a l _s o f t w a r e =factor ( Data $ s t a t i s t i c a l _s o f t w a r e ,l e v e l s = c ( "1 " , " 2 " , " 3 " , " 4 " , " 5 " ) ,ordered = TRUE)
R e s u l t = l i k e r t ( Data )## bar plot

plot ( R e s u l t , type=" bar " )# Example 'R' s c r i p t for the calculation of the# Wilcoxon rank sum t e s t for comparing values# of l i n g u i s t i c ( lin g ) vs . computational ( clcs )# users for question 'Q11_1 'library ( l a t t i c e )q u e s t d a t a = read . csv ( " n u m e r i c a l =Qnb . t x t " )= q u e s t d a t a [ which ( q u e s t d a t a $Q_1 == "EXP" ) ,names ( q u e s t d a t a ) %i n% c ( "Q11_1 " ) ]c l c s <= q u e s t d a t a [ which ( q u e s t d a t a $Q_1 == "LESS_EXP" ) ,ling <

names ( q u e s t d a t a ) %i n% c ( "Q11_1 " ) ]q u e s t d a t a $Q11_1 . f = factor ( q u e s t d a t a $Q11_1 ,ordered = TRUE)
w i l c o x . t e s t ( q u e s t d a t a $Q11_1 ~ q u e s t d a t a $Q_1 )round ( tapply ( q u e s t d a t a $Q11_1 , q u e s t d a t a $Q_1 ,mean , na . rm=TRUE) , d i g i t s =2)