<!-- image -->

## COMPUTING NUMERICAL IMAGES OF STUDENT WRITING: PERSPECTIVES FROM EDUCATIONAL RESEARCH

Morten Tannert Aarhus University mota@edu.au.dk

## Abstract

With the rapid increase in the number of available digital texts in schools, new methodological approaches to studying writing development in education are now emerging. However, with new methodological approaches follow new epistemological challenges. In this article, I examine some of these challenges and discuss how they affect the role of computational linguistics within the field of educational writing research. The article is structured around three main sections. First,  I  position  computational  linguistics  within  the  wider  field  of  educational writing research with particular focus on L1 writing and K12 education. Second, I discuss to what extent methods from computational linguistics can provide us with new  insights  into  different  aspects  of  educational  writing.  Third,  I  discuss  the potential of the concept of affordance to bridge between technology-centered and human-centered  methodological  approaches,  and  I  relate  this  idea  to  recent theoretical  developments  in  the  digital  humanities.  Based  on  this  discussion,  I conclude  the  article  with  suggestions  for  possible  directions  in  future  writing research.

Keywords: Computational linguistics, digital methods, writing research, learner corpora, L1 writing, education

## 1. Introduction

Computational  linguistics  (CL)  is  by  no  means  a  new  research  field.  It traces back to the very early days of computing in the 1940s, and it has since  had  great  impact  within  fields  such  as  machine  translation  and artificial intelligence (Hirst, 2013). It is, however, with the advent of text corpora and corpus linguistics in the 1960s and 1970s (McEnery & Hardie, 2013) and later with the massive digital transformation of our communicative practices in the 21 st century (Baron, 2009) that CL and its related digital methods for processing language have become an important part of many humanistic and social science disciplines. Within educational

## Scandinavian Studies in Language, 12(1), 2021 (1-29)

science, the pervasive digitization of educational practices and  the consequent  stream  of  digital  data  have  resulted  in  an  increasingly  vast number  of  student  texts  now  being  available  in  digital  formats  for researchers.  Unlike  many  other  data  types  generated  by  digital  media (Manovich, 2001, 2020), written texts still hold many  traditional dispositions (e.g., linearity) that make them easier to process computationally. This means that it, at least from a technical perspective, has become easier to build large corpora of texts written by students across all levels of the educational system. As  a result, new  innovative methodological  approaches  for  studying  student  writing  are  emerging, especially from within the field of natural language processing (Eisenstein, 2019), and new sub disciplines within educational science, such as learning analytics (Siemens, 2013) and writing analytics (Moxley et al., 2017), are now seeing the light of day.

With  new  methodological  approaches  follow  new  epistemological challenges, and the 'computational turn' (Berry, 2011) within humanities and  social  science  studies  has  attracted  harsh  critique,  also  from  within educational science (see e.g. Perelman, 2012; Selwyn, 2015, 2019). The critique centers around two main ideas. One, that computational analyses are considered somewhat incompatible with the epistemic pluralism and interpretive processes typically associated with humanistic studies (see e.g. Da, 2019; Fish, 2018a, 2018b). Two, that digital research is often tinged with a touch of technocentrism, in the sense that it is driven more by the possibilities provided by new types of digital data and software tools than by disciplinary agendas (see e.g. Anderson, 2008; van Es et al., 2018; Wise & Shaffer, 2015). These objections are interesting because they touch upon an important question, namely what it actually means to study texts and writing  on  the  basis  of  an  integration  of  humanistic  and  computational approaches.

Responding to these critiques, Gavin (2020) investigates the following question: 'What relation exists between the textual domain and its  numerical image?' (p.4). Gavin's investigation of this question is set within the context of literary studies. He argues that numerical images of texts, which is Gavin's term for the statistical results of computational text analyses,  do  not  merely  decompose  and  reduce  textual  sophistication, rather they reconfigure texts and relate them to other texts within the corpus in ways that are still sensitive to context and subtle semantic nuances. In

## Scandinavian Studies in Language, 12(1), 2021 (1-29)

this  sense,  Gavin  elaborates  on  Moretti's  (2000,  2013)  idea  of  'distant reading',  which,  as  the  name  suggests,  describes  computational  text analysis as an activity in which distance is an epistemological condition; i.e. it allows researchers to examine  units and patterns otherwise indiscernible when close reading texts 'manually', units and patterns that are either much smaller (syntax, morphology, tropes, etc.) or much larger (register, style, genre, etc.) than the text itself.

In computational studies of writing in educational contexts, which is the main topic of this article, the question of how numerical images relate to  different  objects  of  study  (theme,  genre,  register,  etc.)  is  even  more complex. This complexity is first and foremost due to the normative nature of education, where different modes of evaluation, among other important elements, are constitutive of educational practices (Bernstein, 1990). This constitution means that not only must the numerical images generated by computational analyses represent the text itself, but they must also act as a performance proxy of other more pedagogically inherent phenomena, such as student competence or school discourse. This means a further inflation of the cross-disciplinarity of CL in the sense that computing and linguistics must  also  adhere  to  principles  from  other  disciplines  such  as  cognitive psychology and psychometrics.

In this article, I examine the role of CL within the field of educational writing research. To do so, I first position computational studies of writing within the broader field of writing research in education. I then discuss to what extent CL methods can provide us with new insights into different aspects of educational writing. This discussion is based partly on a model of  an  educational  writing  situation  in  which  I  elaborate  on  a  four-part language model proposed by Stubbs (2007) and partly on Carter's (2007) metagenres for educational writing. Finally, I discuss whether the notion of affordance provides a viable conceptual pathway for CL to bridge between technology-centered and human-centered approaches to studying writing development in education.

## 2. Educational writing research

Since the introduction of the notion of multiliteracies in the seminal work of The New London Group (1996), much theoretical and empirical inquiry within education  has  examined  the  new  networked  communicative practices caused by digital media, with particular emphasis on multimodality (e.g. Cope & Kalantzis, 2000, 2009; Jewitt, 2006; Lankshear & Knobel, 2003).  Drawing  on  social  semiotic  ideas  first  introduced  by Halliday (1978), the notion that contemporary communication works by means  of  many  different  communicative  modes  has  become  dominant, especially  pioneered  by  the  work  of  Kress  &  van  Leeuwen  (1996)  and Kress  (2010).  Alongside  the  focus  on  multiliteracies,  much  writing research within the last two decades has focused on the wider sociocultural dimensions of writing (Barton, 1994; Street, 1984, 1995), such as matters of writer identity and writing instruction (Parsons et al., 2020). The result of  this  focus  has  been  that  sociolinguistic  and  ethnographic  approaches have dominated the field. However, forwarded particularly, of course, by the appearance of new digital methods, but also by the fact that written language still remains the main mode of non-verbal communication in both school and everyday life (Biber & Egbert, 2018; Brandt, 2014; Bremholm et al., 2018), we are now slowly seeing a resurgence of written language itself as an important object of study in writing research.

## 2.1 Corpus-based studies of L1 writing

It has long been common within fields such as second language acquisition (L2) or foreign language teaching to study corpora of learner language. Studies of, for instance, language transfer, specific linguistic features or larger rhetoric constructs are thus seen as key in understanding different stages of interlanguage development or as a basis for instructional decisionmaking (see e.g. Reppen, 2010; Tracy-Ventura & Paquot, 2020). However, if we look at studies focusing exclusively on L1 writing, the image is less clear. Influenced by, among others, the works of Biber (2006) and Hyland (2004),  several  L1  studies  have  focused  on  different  dimensions  of language use in higher education, such as register variation, stance or voice. Most of these studies have targeted professional and published academic writing in the form of, for instance, journal articles and textbooks (see e.g. Gray, 2015; Römer et al., 2020) or texts written by university students (see e.g. McNamara et al., 2009). K12 i , on the other hand, is still a relatively understudied context. Two important reasons for this have probably been that texts have been difficult to collect due to a lack of standardized digital infrastructure in schools, and that primary and lower secondary writing is more erroneous and thus complicates pre-processing of textual data as well as the data analysis itself. As a result, corpus-based studies of student texts from K12 are often based on prompted writing (as opposed to naturally occurring writing) and thus presuppose an intervention-based or experimental research design.

## 2.2 Diachronic and synchronous approaches

Although overlapping and entangled, K12 studies can for the sake of clarity be divided into two groups: those that adopt a diachronic perspective and those  that  adopt  a  synchronous  perspective.  Diachronic  studies  are typically longitudinal and often focus on developmental aspects of either different types of writing (narrative, persuasive, argumentative etc.) (see e.g. Beard & Burrell, 2010) or on specific genres (see e.g. Olinghouse & Wilson,  2013).  Many  of  these  studies  adopt  a  functionalist  linguistic perspective,  often  originating  in  Halliday's  (2013)  systemic  functional linguistics. Christie & Derewianka (2010), for instance, studied disciplinary  and  genre  related  differences  in  the  writing  trajectories  of children aged 5-18. Others, such as Troia et al. (2019) and Beers & Nagy (2011), have studied how different word and syntax level variables develop as a function of grade level. A shared feature in these studies is an interest in studying linguistic development in student texts across age levels using either a longitudinal or a cross-sectional design.

Synchronous studies, which likely outnumber diachronic studies, are more focused on immediate differences between texts in a comparative or evaluative  perspective.  Hardy  &  Römer's  (2013)  study  of  disciplinary differences  in  upper-level  student  writing  is  a  good  example  of  this approach.  In  this  study,  the  authors  show  how  different  registers  are realized linguistically in student texts and examine how the registers vary according to disciplines. The purpose of synchronous studies is thus not to study how writing develops across time or year levels, but rather to study how texts differ according to synchronous variables (e.g. genre, motivation, writing attitude, or self-efficacy). Another important research area  within  this  approach  is  writing  assessment,  particularly  automated essay scoring, which is a specific kind of automatic assessment based on NLP  and  machine  learning  techniques  (Shermis,  2014;  Shermis  & Burstein, 2013). Studies in this area aim to examine the characteristics of texts  with  different  proficiency  scores,  which  can  then  be  used  to automatically assign other texts with a certain score.

## 3. What is a student text?

Before discussing more directly the potential of CL methods in educational writing  research,  it  is  important  to  consider  a  couple  of  rhetorical circumstances  that  make  educational  writing  very  different  from  other types of writing.

## 3.1 Student texts as text acts

From a rhetorical perspective, student texts are complex. On the one hand, they  are  socially  situated  textual  acts ii between  a  student  and  a  teacher within a confined educational context with more or less clear social and textual  norms.  But,  on  the  other  hand,  they  are  also  fictional  texts addressing fictional readers outside the context of the textual act (Togeby, 2015).  This  means  that  the  success  criteria  of  student  texts  do  not necessarily follow from the genre, as they would otherwise if we followed, for instance, Swales' (1990) genre definition. For instance, writing a job application  in  school  does  not  have  employment  as  an  actual  success criteria; instead, the text hast to meet metafunctional success criteria such as adhering textual norms explicated by the teacher or getting a specific grade  (Berthelsen,  2021).  Of  course,  this  ambiguity  in  the  rhetorical situation manifests itself in different ways in both writing assignments and student texts (Juuhl, 2020; Kvistad & Otnes, 2019; Troelsen, 2018), but the general idea is that the illocutionary force in student texts, i.e. the intended function of an utterance (Searle, 1969), is either suspended or at least to some extent distorted.

This aspect of educational writing is in many ways deeply embedded in the disciplinary structure of schooling. As pointed out by Carter (2007), the textual act of writing in education expresses different kinds of knowing and doing within the disciplines, because the primary purpose of writing within  a  discipline  is  learning,  and  the  successful  realization  of  such  a purpose is of course related to - but not directly equivalent to - a generic notion of writing competence or text quality. Instead, writing in different subjects  relates  to  different  kinds  of  academic  situations  that  require different  ways  of  expressing  knowledge  and  doing.  Therefore,  Carter proposes four metagenres for educational writing, which relate to different disciplinary traditions. The first metagenre, problem-solving , is related to disciplines originating in the applied and project-oriented sciences. It is a kind of writing aimed at identifying  real-world problems, analyzing the

## Scandinavian Studies in Language, 12(1), 2021 (1-29)

problem  using  theory,  suggesting  possible  solutions,  and  evaluating  the project according to established criteria.   The second metagenre, empirical inquiry , is aimed at examining phenomena based on systematic investigations of empirical data. The first two metagenres have relatively generic success criteria known from the sciences (correct use of methods, truth value, practical impact, etc.), which function as communicative ends in  themselves.  That  is,  however,  not  the  case  with  the  third  metagenre, research from sources . According to Carter, a distinguishing feature of the third  metagenre  is  that  the  value  of  the  textual  act  is  intrinsic  to  the discipline. It involves learning to write based on other sources, not as an end  in  itself,  but  as  a  means  of  learning  a  specific  disciplinary  kind  of knowing. Hence, when students in school are asked to write an analysis of a  novel,  for  instance,  the  value  of  the  textual  act  is  not  primarily  about whether they have followed an explicit procedure correctly or whether their analysis is true (as it would be if they were conducting an experiment in biology); it is instead about learning to understand literature from a specific disciplinary perspective. In the fourth metagenre, performance , the focus is on the act of performing and specifically on the result (artifact, text, etc.) of such a performance. An example of this is when students are asked to write texts for no other purpose than showing the teacher that they master specific textual norms or writing styles. The four metagenres are summed up in figure 1 below.

As has no doubt become clear, student writing is in many ways a complex  compound  of  rhetorical,  social,  institutional  and  disciplinary dimensions.  The  rhetorical  ambiguity  is  common  for  most  educational writing, but, as displayed by Carter's (2007) metagenres, it manifests itself differently in different disciplines. This obviously has implications for how to  conceive  student  texts  as  research  data,  which  I  will  examine  in  the following section.

Table 1: Carter's (2007) metagenres for educational writing

| Metagenres (Carter, 2007)   | Characteristics                                                                                                                                                                                                         | Examples of educational writing tasks                                                                 |
|-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------|
| Problem-solving             | Identifying real-world problems, analyzing problems based on theory, suggesting solutions, evaluating according to established criteria. Success criteria: Correct use of methods and viability of suggested solutions. | STEM reports Social studies papers Project reports                                                    |
| Empirical inquiry           | Formulating hypotheses, collecting empirical data, systematically investigating empirical data. Success criteria: Correct use of methods and validity of results.                                                       | Biology reports Physics/chemistry reports                                                             |
| Research from sources       | Examining phenomena based on established sources, discussing. Success criteria: Adopting and becoming familiar with a specific kind of disciplinary knowing.                                                            | Literary analyses Cultural essays Historical essays                                                   |
| Performance                 | Performing according to specific norms and criteria, the product is evaluated according to different levels of proficiency. Success criteria: Displaying proficiency according to                                       | Non-academic texts (e.g. journalistic texts and job applications) Essays in foreign language teaching |

## 4. Student texts and their numerical images

In  the  introduction,  I  presented  the  idea  that  the  numerical  images generated by CL can have multiple referents related to linguistic, cognitive and wider social and cultural dimensions, e.g. discourse, register variation or  individual  competence.  In  the  following,  I  will  therefore  discuss  the potentials and limits of using methods from CL to study different aspects of educational writing. I will first introduce a model representing a reduced graphical  configuration  of  an  educational  writing  situation.  I  will  then consider the individual parts of the model relating each of them to Carter's metagenres and discussing to what extent CL methods can provide new epistemological insights into the individual dimensions.

## 4.1 Model of an educational writing situation

Educational writing research is an umbrella term for the study of a range of different concepts. Concepts such as school discourse, writing competence, academic voice and adverbial phrases all share the notion that they can been studied by means of text corpora, but they are also distinct from each other and hold distinctly different ontological properties. When studying  text  corpora  we  are,  on  the  one  hand,  studying  observable utterances, e.g. words and sentences, that exist as brute facts (Searle, 1995) with physical properties (e.g. as ink marks on a piece of paper or as pixels on  a  screen),  although  corpus-based  studies  of  texts  can  of  course  not merely be reduced to studying such physical aspects of language. We are, on  the  other  hand,  also  studying  something  else,  namely  the  social, structural  and  cognitive  dimensions  related  to  specific  instances  of language use. Stubbs (2007) thus argues for a pluralist model of language as  the  basis  of  corpus  studies.  Drawing  on  Tuldava's  (1998)  original distinctions, Stubbs proposes a four-part model  consisting of two overlapping  conceptual  pairs.  The  first  is  between  the  potential  and  the actual, i.e. between the prerequisites and constraints of language use (e.g. individual competence or what is possible within the language system) and actual  realized  language  use  (e.g.  discourse  or  text),  and  the  second  is between  process  and  product,  i.e.  between  the  process  of  producing language (e.g. writing a text) and the linguistic products emerging from these processes (e.g. the text itself). To varying extents, it is possible to study these elements computationally.

To fully understand educational writing, however, I argue that we need a more specialized model. In particular, the concept potential needs to be specified not only in terms of grammar and the cognitive capacities of  the  individual,  but  just  as  importantly  in  terms  of  the  social  and pedagogical dimensions that form specific instances of educational writing.

Student Sociocultural framing Instructional framing Writing process Several  studies  have,  for  instance,  shown  that  student  writing  is  highly susceptible to different instructional framings (see e.g. Graham et al., 2012; Graham  &  Perin,  2007)  and  to  individual  variables  such  as  social background (Dunsmuir & Blatchford, 2004) and writing attitude (Graham et  al.,  2017).  In  the  model  below,  I  therefore  distinguish  between  the instructional framing and the sociocultural framing of educational writing. Other more specialized and conceptually complex models have of course been developed, such as Flower & Hayes' (1981) cognitive process model or  Kern's  (2000)  contextual  literacy  model.  These  models  all  provide valuable insight into different dimensions of writing. However, for the sake of the arguments made in this section, it is sufficient to provide a minimalist graphical display representing a reduced configuration of an educational writing situation.

Figure 1: Minimal graphical configuration of an educational writing situation

<!-- image -->

The center  of  the  model  shows  the  student  (i.e.  the  writer),  the  writing process  and  the  student  text,  which  mirror  Stubbs'  (2007)  notions  of potential , process and actual/product respectively. In this case, the student refers  to  the  cognitive  capacities  of  the  individual  writer,  the  writing process to the process of producing written text, and the student text to the actual  physical  record  of  written  language.  As  briefly  hinted  above, instructional framing refers to the specific instructional designs that form the  writing  situation,  such  as  the  writing  task  or  scaffolding  activities, whereas sociocultural framing refers to social variables related to both the individual,  such  as  social  background  or  writing  habits,  and  to  wider cultural and institutional aspects, such as school discourse or disciplinary traditions.

## 4.2 What computational linguistics can and cannot tell us about educational writing

CL  deals  with  linguistic  data,  often  in  the  form  of  written  texts  or transcriptions of spoken language. In the following section, I will discuss to what extent computational analyses of this kind of data can provide us with insights into different aspects of educational writing.

## 4.2.1 The student

In this context, studying 'the student' means to study individual competence. When student texts are used to study individual competence, they are essentially treated as performance data, i.e. as data that represents deliberate  student  actions  relating  to  qualitatively  different  levels  of proficiency.  However,  as  we  have  seen  in  the  description  of  Carter's metagenres, students display competence in different ways according to different purposes and success criteria in each metagenre. This means that the extent to which these success criteria can be measured computationally varies  according  to  metagenre.  An  obvious  aim  in  educational  writing research is, for example, to study students' writing competence, either as a generic construct or as a more domain-specific or genre-specific skill (e.g. narrative writing or argumentative writing). This relates directly to Carter's metagenre performance in  which  the  primary  success  criteria  of  student text acts are not related to communicative purposes, but rather to displaying mastery  of  specific  linguistic  structures  and  norms.  This  is  of  course particularly  evident  in  foreign  language  teaching,  where  students  often write to practice specific linguistic structures and styles, but it also applies to L1 writing, which is often assessed on the basis of formal criteria, such as correctness or style.

An  important  research  area  aimed  at  measuring  the  individual competence of students by using CL methods is that of writing assessment. In recent years, the application of CL methods has been particularly evident within the field  of  automated  essay  scoring,  in  which  researchers  try  to develop software tools that can automatically analyze features in texts and on that basis assign similar texts with a certain score (Correnti et al., 2020; Shermis, 2014). The theoretical underpinnings of many of these studies are often located in probabilistic inference or more widely within the field of psycholinguistics  (Bod,  2009;  Jurafsky,  2002).  A  highly  debated  and disputed idea within these fields regards the relation between performance and  competence.  Since  the  introduction  of  the  Chomskyan  dualism between 'e-language' and 'i-language' (Chomsky, 1965, 1966), it has been debated how these concepts are related - if at all - and whether performance is  a  valid  proxy  for  an  individual's  mental  competence  to  produce language. However, as Leech (1992) points out, these quarrels are often overemphasized,  and,  although  the  Chomskyan  notion  in  particular  has challenged corpus linguistic studies, there is a consensus in many corpus linguistic  research  environments  that  performance  and  competence  are somehow  causally  related  in  the  sense  that  performance  is  a  product, though not a 1:1 reflection, of individual competence.

If we turn to Carter's remaining metagenres, it is clear that it is much more difficult to measure competence within these genres by only looking at textual data. A common feature of these metagenres is that their success criteria are related to aspects that rely on human judgement. A key success criterion for the metagenre problem-solving is, for instance, the viability of the suggested solutions, and, similarly, the success criterion for empirical inquiry is  the  validity  of  results.  If  competence  in  such  areas  is  to  be measured, one would need scales based on human scoring, such as rubrics. The metagenre performance is thus the only genre in which language is an end in itself, and it is therefore much easier to study competence within this area by taking measurements of linguistic features in the text as evidence.

## 4.2.2 The writing process

Although studies of writing processes are slowly becoming more frequent in writing research, they are still relatively rare. Most of these studies do not rely on textual or linguistic data, but rather on qualitative analyses of alternative data sources such as classroom observations or screen recordings or on small quantitative analyses of rubric scores or log data. Almond et al. (2012), Leijten et al. (2015), and ten Peze et al. (2021), for instance, examine keystroke log data to study how students use different strategies for revising their texts while writing, and Engblom et al. (2020) use screen recordings from the students' computers to study the types of changes students make to their texts when writing. This shows that it is difficult  to  study  writing  processes  as  they  unfold  simply  by  measuring linguistic features in texts; we must instead rely on a larger collection of different data sources.

## 4.2.3 The student text

It can of course be difficult to distinguish the student text from the other aspects of the model, such as sociocultural framing or the student. The main purpose of traditional corpus linguistic studies, however, has often been to explore  recurring  linguistic  patterns  in  language  use.  These  studies  are conducted  to  inform  linguistic  theories  (text  linguistics,  morphology, semantics etc.) rather than pedagogical theories of language learning, and they thus make new connections between language as an abstract system (langue) and actual patterns in language use (parole) (Biber et al., 1998; Stubbs,  2007).  Such  purely  linguistic  studies  are  rare  when  studying student texts in educational writing, because the primary endeavor of much educational  research  is  often  problem-oriented  or  aimed  at  developing educational practice, and linguistic measures are therefore often related, at least indirectly, to phenomena outside the text itself. This means that the student text is often only interesting insofar as it works as a proxy for other more inherently pedagogical phenomena.

An  important  educational  issue  related  directly  to  the  text  itself, however, is that of text quality. A general notion of text quality is difficult to infer directly from computational analyses of texts alone, because we have no parameters for assessing what an appropriate level of syntactic complexity is, for instance. This will inevitably vary from one communicative context to another. One problem is, for instance, that texts can be manipulated and twisted in a way that turn them into meaningless strings  of  words  while  still  maintaining  the  same  linguistic  surface structures. A famous example is the BABEL (Basic Automatic BS Essay Language)  generator 1 developed  in  2014,  which,  based  on  a  limited

1  The BABEL generator can be found here: https://babel-generator.herokuapp.com/

## Scandinavian Studies in Language, 12(1), 2021 (1-29)

number  of  keywords,  was  able  to  generate  gibberish  essays  that  were assigned high scores by a number of machines for automatic essay scoring (Perelman,  2020).  Another  problem  is  that  text  quality  in  educational contexts is often not derived from functional communicative criteria but rather from metafunctional criteria such as grading or, in Carter's (2007) terms, displaying specific disciplinary kinds of knowing and doing. This means  that  many  measures  of  text  quality  rely  on  holistic  scores  (e.g. grades  or  rubric  scores)  rather  than  on  measures  of  specific  linguistic features. This does not mean, however, that measures of linguistic features are completely separated from a general idea of text quality. Many studies have, for instance, examined which linguistic features predict text quality (see Crossley (2020) for an overview), but as pointed out by, among many others, Beck & Jeffery (2007) and Murphy & Yancey (2007), the features that constitute text quality are relative to genre, task, context and several other  variables.  Thus,  although  we  can  easily  calculate  which  features predict text quality and thus create an aggregate and probabilistic measure of quality, it is much more difficult to calculate text quality of individual texts based on linguistic measures alone.

## 4.2.4 Instructional framing

As was the case with the writing process, the instructional framing is of course difficult to study directly in student texts without additional data sources.  However, it is  common in corpus linguistic studies to examine which  contextual  factors  account  for  variability  in  student  texts,  which means that it is possible examine how a specific instructional design affects linguistic  features  in  student  texts.  Rousse-Malpat  et  al.  (2019),  for instance,  studied  how  two  different  instructional  approaches  affected morphosyntactic  and  lexical  features  in  L2  students'  texts,  and  a  metaanalysis by Graham et al. (2012) showed general effects of instruction on different dimensions of writing.

A second research area that relates to instructional framing concerns teacher feedback. Written feedback in the form of either text comments or longer segments of prose is available in digital formats, and, although this practice is still in the very early stages, researchers have been able to create corpora of written teacher feedback. This means that it is possible to study feedback practices by applying tools from natural language processing or other digital tools to study patterns in teachers' feedback practices (see e.g.

Lang, 2018). Another potential application of CL methods within this areas is  not  concerned  with  written  language  as  such  but  rather  with  written records of spoken language, such as classroom dialogue (O'Keeffe et al., 2007). However, at the moment, these studies are very rare.

## 4.2.5 Sociocultural framing

The final aspect of the model is sociocultural framing, which encompasses both  sociocultural  (e.g.  social  background  and  writing  attitude)  and linguistic variables (e.g. register, genre, and voice). As to the former, the limits  are  similar  to  those  previously  described  with  regards  to  writing process and instructional framing. To study such aspects successfully, we would require more data sources than mere text. With regard to the latter, however, it is different. Large linguistic constructs, i.e. constructs that are shared within a given language community, can of course be studied by means  of  corpora.  Discourse,  genre  and  register,  for  instance,  might manifest themselves differently for each metagenre, but the possibility of studying these constructs is independent of metagenre; in other words, we can study them in all metagenres by only looking at the textual data.

## 4.3 Implications for writing research

As the above examination shows, the potential of CL methods in writing research is a delicate matter, because it differs from one metagenre to the other.  It  is  a  widespread  approach  to  infer  from  linguistic  features  to cognitive constructs and  to those sociocultural  dimensions  that  are inherently sociolinguistic. However, important dimensions such as writing process and instructional framing are omitted in this process. This clearly has consequences for how to understand the role of CL in writing research.

First of all, the above examination unveils a significant disciplinary imbalance. Those disciplines that make use of the performance metagenre are more easily compatible with CL methods than those that make use of, for  instance,  the problem  solving or empirical  inquiry metagenre.  By 'compatible', I mean that it is easier to measure writing competence in the performance metagenre by only taking measurements of linguistic features as  evidence.  The  other  metagenres  have  success  criteria  that  are  extralinguistic  and  thus  often  rely  on  some  kind  of  human  scoring.  The limitations  of  CL  methods  in  terms  of  addressing  these  non-linguistic dimensions thus suggest a need for a multifaceted approach more attuned to  the  complexity  of  educational  writing.  This  means  an  approach  that focuses on a wider range of data types within an overarching educational ecology,  an  approach  that  focuses  on  what  Palermo  (2017)  refers  to  as Writing as Analytics , i.e. writing as socio-cognitive, scripted and transformative  processes.  Such  an  approach  would  also  address  another significant issue, namely that contemporary digital texts consist not only of linguistic structures but also of a range of other communicative resources such as pictures, layout and colors (Kress, 2010; Kress & van Leeuwen, 1996). Layout, for instance, has proven to be a particularly important area of young children's writing development (Kress, 1994). Linguistic studies must therefore be supplemented with studies of multimodal corpora if we are to gain a better insight into the writing trajectories of school students.

## 5. Moving the discussion forward

Although increasingly prevalent, the potential of CL methods and tools is still  relatively  unexplored  in  educational  writing  research.  As  I  briefly touched upon in the introduction,  the  application  of  CL  methods  in  the humanistic and social sciences raises several important questions. For one thing, it raises the question of how digital tools affect knowledge generated within  educational  writing  research,  and,  more  fundamentally,  what  it means to study writing on the basis of an integration of humanistic and computational approaches. These are important questions to address, since digital  tools  are  likely  to  play  an  increasingly  important  role  in  future writing research. It is, however, beyond the limitations of this article to give a detailed account or in-depth discussion of these questions. Instead, I will briefly identify a few theoretical concepts that could provide entry points for further inquiry into these matters.

## 5.1 Affordances of digital research tools

The first theoretical entry point concerns the role of research instruments. To fully grasp the potential of CL methods and tools, one must take into the account the nature of the research instruments themselves. As pointed out  by  Baird  (2004)  in  his  materialist  account  of  scientific  knowledge, scientific  instruments  are  not  merely  instrumental  to  the  production  of scientific  knowledge.  They  are  also  constitutive.  This  means  that,  in different ways, they affect the epistemological underpinnings of different scientific  disciplines,  because  they  make  certain  types  of  knowledge

## Scandinavian Studies in Language, 12(1), 2021 (1-29)

available and other types of knowledge unavailable. In parts of the natural sciences  this  is,  for  instance,  connected  to  the  physical  properties  of research instruments (e.g. the thermodynamics of mercury in a thermometer). When dealing with digital technologies, however, it is of course insufficient to just take into account the physical properties of the research instruments. Digital technologies also allow for an array of social and communicative actions to be carried out digitally, i.e. by virtue of the digital software of the computer (Berthelsen & Tannert, 2020). This means that, in the case of NLP tools, for instance, there is a digital 'materiality' that  affects  what  actions  one  can  and  cannot  perform  by  means  of  the specific tool, which again affects what kind of knowledge is produced.

A way of understanding more thoroughly what this means is to turn to the concept of affordance, which was originally coined by Gibson (1979) in his classical theory of visual perception. Affordances denote possibilities for action (e.g. social, communicative, and aesthetic actions) enabled by the materiality of the physical or virtual environment relative to context and agent; in other words, affordances are not fixed and generic properties of the environment but differ according to who is using them and under which circumstances they are used. The term has been applied to a range of  different  domains,  particularly  within  the  field  of  technology  studies (Hutchby, 2001). In all  of these cases, affordance is used to investigate properties in the physical and digital environment  of the specific technology  that  make  certain  types  of  action  possible.  It  is  therefore possible to also use the concept to refer to the different possibilities for action brought about by digital research tools. The notion of affordance has been sporadically used in research literature to refer to the general idea of 'potentials' in research tools. Kyle (2021), for instance, uses the term to investigate the potentials of specific NLP tools for learner corpus research, while Dobson (2019) uses the term in a somewhat wider sense to critically examine  possibilities  and  limitations  of  computational  methods  in  the humanities and social sciences at large. However, if used in a too general sense an important element of affordances get lost, namely that affordances are  not  generic  properties  but  are  relative  to  agent  and  environment.  In other  words,  they  are  purposefully  utilized  by  an  agent  with  certain dispositions  in  order  to  carry  out  an  intended  action  within  a  specific context. Thus, affordances can be used as a theoretical basis for examining the epistemic potentials afforded by different digital tools in relation to very specific scientific disciplines and research questions.

An important aspect of affordances is that they ascribe agency to the individuals utilizing them. This means that it is not sufficient to only focus on the tools as isolated technologies; they must instead be viewed as part of a larger research process involving traditional humanistic approaches. This idea leads me to the second theoretical entry point, one that concerns the role of the researcher in the process of interpreting numerical data.

## 5.2 Computational hermeneutics and data narration

There is of course no single unified epistemological approach for studying writing. No theoretical perspective covers writing in all of its complexity and different manifestations, and there is no single method (quantitative nor qualitative) or research instrument for adequately describing writing. Consequently,  single  linguistic  measures  must  be  invigorated  by  being compared and countervailed with other measures or data types before they become part of a broader process of interpretation. The reliance on multiple data  sources  and  interpretational  processes  has  therefore  become  an important  topic  in  the  literature  on  both  writing  analytics  and  digital humanities in general - particularly with a focus on the epistemological relation  between  computational  analysis  and  hermeneutic  interpretation, or, in other words, what has become known as computational hermeneutics.

A useful theoretical contribution in that area is the three types of text reading described by Mohr et al. (2015); thin reading, close reading and thick  reading.  Thin  reading  refers  to  the  process  of  content  analysis  in which  the  distribution  of  key  meanings  through  texts  in  a  corpus  is identified  and  turned  into  numerical  data  operable  by  computers.  This process thus groups text in an attempt to find general patterns of manifest meaning without paying attention to subtle semantic differences. This is, however,  the  case  with  traditional  manual  close  reading  in  which  the complexity and peculiarities of individual texts are identified and interpreted by human readers, not by computers. With thick reading, on the other hand, the thrust once again shifts towards computational analyses, but this time not in order to reduce complexity but rather to focus on utilizing as many meaningful measures as possible and thereby creating a richer data material that fits one's interpretive intention. It is thus an attempt to bring computational analyses closer to the act of close reading by integrating the scale  of  data  material  in  thin  reading  with  the  epistemic  plurality  and textual sensitivity of close reading. This kind of quantitatively based 'thick description' does of course not speak for itself but will need to become part of  a  larger  interpretational  narrative.  Arguing  against  the  idea  that  data analyses are self-evidentiary,  Dourish & Cruz (2018) write: 'Data makes sense only to the extent that we have frames for making sense of it, and the difference  between  a  productive  data  analysis  and  a  random-number generator is a narrative account of the meaningfulness of their outputs.' (p. 8).  Narrating  data  entails  structuring  and  interpreting  data  within  a theoretical frame that supports a more thorough understanding of the data and the analytical outputs. Thus, it is this kind of data-rich and multifaceted approach that would also allow CL studies of writing to merge traditional computational approaches involving abstraction and automation with wellknown humanistic approaches involving contextualization and interpretation.

## 6. Concluding remarks

This article has offered an explorative discussion of the role and potential of  CL  methods  in  educational  writing  research.  In  one  sense,  this  is  a discussion  that  is  particular  for  writing  research,  in  the  sense  that  it concerns  matters  such  as  linguistic  representation,  writing  development and  pragmatics.  In  another  sense,  however,  the  discussion  of  CL  and writing research is subsidiary to more general pedagogical concerns about the  perpetually  scrutinized  relation  between  education  and  its  related scientific  disciplines.  An  immediate  concern  here  is  of  course  whether applications of CL methods in educational writing research will lead to a reduced understanding of writing by ignoring the broader social contexts in  which  writing  practices  unfold,  and,  consequently,  whether  such  a reduced understanding of writing will contribute to an already increasingly reductionist  understanding  of  pedagogical  practice  and  the  purposes  of education as such (Biesta, 2015; Selwyn, 2015). What we have learned already from introducing fields such as learning analytics into educational research is their potential to cause a great divide between data science and other areas of educational research, such as the philosophical or anthropological areas. This divide stems - at least to some extent - from the uncritical promise of prosperity that sometimes comes with data science, such as promising quick-fixes to long-lasting educational problems. It is perhaps more fruitful to engage in specific discussions on what data can and cannot say in relation to very specific research questions. This also applies  to  the  discussions  on  CL.  Educational  writing  is  not  a  uniform practice.  It  relates  to  very  different  disciplinary  situations  and  contains different communicative criteria, and the discussions that we ought to have on the potentials of CL for studying writing should pay more attention to this heterogeneity.

In this article, I have therefore attempted to highlight the need for an approach to writing research that accommodates critiques of reductionism and technocentrism by bridging the gap between technology-centered and human-centered approaches for studying writing. It is of course nothing new to claim that data analysis and interpretation go hand in hand, but this perspective is often lacking in many discussions on the potential of CL and data science at large, particularly in discussions on policymaking (Berthelsen & Tannert, 2019). This means that too much emphasis is put on the data itself without paying sufficient attention to how this data can be meaningfully  interpreted and  made  sense  of  within an educational framework, which by nature is normative and contested. Such aspects are, however, vital to consider further if we are to successfully integrate CL methods in educational writing research.

## Notes

i K12 is an American collective term for all the educational stages from kindergarten to the 12 th grade.

ii A text act is Togeby's (2010) term for a series of related sentences (a text) that constitute one coherent act, rather than a series of independent speech acts.

## References

Almond, R., Deane, P., Quinlan, T., Wagner, M., & Sydorenko, T. (2012). A Preliminary Analysis of Keystroke Log Data from a Timed Writing Task. ETS Research Report Series , 2012 (2), 1-61. https://doi.org/10.1002/j.2333-8504.2012.tb02305.x

Anderson, C. (2008). The End of Theory: The Data Deluge Makes the Scientific Method Obsolete. Wired . https://www.wired.com/2008/06/pb-theory/

Morten Tannert Scandinavian Studies in Language, 12(1), 2021 (1-29)

- Baird, D. (2004). Thing knowledge: A philosophy of scientific instruments . University of California Press. https://doi.org/10.1525/j.ctt1pngk6
- Baron, D. (2009). A Better Pencil: Readers, Writers, and the Digital Revolution . Oxford University Press.
- Barton, D. (1994). Literacy: An introduction to the ecology of written language . Blackwell.
- Beard, R., & Burrell, A. (2010). Investigating narrative writing by 9-11year-olds. Journal of Research in Reading , 33 (1), 77-93. https://doi.org/10.1111/j.1467-9817.2009.01433.x
- Beck, S. W., & Jeffery, J. V. (2007). Genres of high-stakes writing assessments and the construct of writing competence. Assessing Writing , 12 (1), 60-79. https://doi.org/10.1016/j.asw.2007.05.001
- Beers, S. F., & Nagy, W. E. (2011). Writing development in four genres from grades three to seven: Syntactic complexity and genre differentiation. Reading and Writing , 24 (2), 183-202. https://doi.org/10.1007/s11145-010-9264-9
- Bernstein, B. (1990). The Structuring of Pedagogic Discourse . Routledge.
- Berry, D. (2011). The computational turn: Thinking about the digital humanities. Culture Machine , 12 , 1-22.
- Berthelsen, U. D. (2021). Digitale tekster og skriftlig fremstilling i gymnasiet-Et curriculumperspektiv. Tidsskriftet Læring Og Medier (LOM) , 13 (23), 1-17.

https://doi.org/10.7146/lom.v13i23.120963

- Berthelsen, U. D., & Tannert, M. (2019). The Ecology of Analytics in Education: Stakeholder Interests in Data-Rich Educational Systems. International Journal of Learning Analytics and Artificial Intelligence for Education (IJAI) , 1 (1), 89-101. https://doi.org/10.3991/ijai.v1i1.11023
- Berthelsen, U. D., & Tannert, M. (2020). Utilizing the affordances of digital learning materials. L1 - Educational Studies in Language and Literature , 20 , 1-23. https://doi.org/10.17239/L1ESLL2020.20.02.03
- Biber, D. (2006). University language: A corpus-based study of spoken and written registers . John Benjamins Publishing Company.

## Scandinavian Studies in Language, 12(1), 2021 (1-29)

- Biber, D., Conrad, S., & Reppen, R. (1998). Corpus Linguistics: Investigating Language Structure and Use . Cambridge University Press.
- Biber, D., & Egbert, J. (2018). Register Variation Online . Cambridge University Press.
- Biesta, G. (2015). Improving education through research? From effectiveness, causality and technology to purpose, complexity and culture. Policy Futures in Education , 14 (2), 194-210. https://doi.org/10.1177/1478210315613900
- Bod, R. (2009). Probabilistic Linguistics. In B. Heine & H. Narrog (Eds.), The Oxford Handbook of Linguistic Analysis (pp. 663-692). Oxford University Press. 10.1093/oxfordhb/9780199677078.013.0025
- Brandt, D. (2014). The Rise of Writing: Redefining Mass Literacy . Cambridge University Press. https://doi.org/10.1017/CBO9781316106372
- Bremholm, J., Hansen, R., & Slot, M. F. (2018). Student work and the multimodal challenge: A mixed methods study of students' productive work in L1, science and mathematics. L1 - Educational Studies in Language and Literature , 18 , 1-27. https://doi.org/10.17239/L1ESLL-2018.18.01.12
- Carter, M. (2007). Ways of Knowing, Doing, and Writing in the Disciplines. College Composition and Communication , 58 (3), 385-418.
- Chomsky, N. (1965). Aspects of the theory of syntax . MIT Press.
- Chomsky, N. (1966). Cartesian Linguistics: A Chapter in the History of Rationalist Thought . Harper & Row.
- Christie, F., & Derewianka, B. (2010). School discourse: Learning to write across the years of schooling . Continuum.
- Cope, B., & Kalantzis, M. (2000). Multiliteracies: Literacy learning and the design of social futures . Routledge.
- Cope, B., & Kalantzis, M. (2009). 'Multiliteracies': New Literacies, New Learning. Pedagogies: An International Journal , 4 , 164-195. https://doi.org/10.1080/15544800903076044
- Correnti, R., Matsumura, L. C., Wang, E., Litman, D., Rahimi, Z., & Kisa, Z. (2020). Automated Scoring of Students' Use of Text

## Scandinavian Studies in Language, 12(1), 2021 (1-29)

Evidence in Writing. Reading Research Quarterly , 55 (3), 493520. https://doi.org/10.1002/rrq.281

- Crossley, S. (2020). Linguistic features in writing quality and development: An overview. Journal of Writing Research , 11 (3), 415-443. https://doi.org/10.17239/jowr-2020.11.03.01
- Da, N. Z. (2019). The Computational Case against Computational Literary Studies. Critical Inquiry , 45 (3), 601-639. https://doi.org/10.1086/702594
- Dobson, J. E. (2019). Critical Digital Humanities: The Search for a Methodology . University of Illinois Press.
- Dourish, P., & Cruz, E. G. (2018). Datafication and data fiction: Narrating data and narrating with data. Big Data & Society , 5 (2), 1-10. https://doi.org/10.1177/2053951718784083
- Dunsmuir, S., & Blatchford, P. (2004). Predictors of writing competence in 4- to 7-year-old children. British Journal of Educational Psychology , 74 (3), 461-483. https://doi.org/10.1348/0007099041552323
- Eisenstein, J. (2019). Introduction to natural language processing. The MIT Press.
- Engblom, C., Andersson, K., & Åkerlund, D. (2020). Young students making textual changes during digital writing. Nordic Journal of Digital Literacy , 15 (03), 190-201.

https://doi.org/10.18261/issn.1891-943x-2020-03-05

- Fish, S. (2018a). If You Count It, They Will Come. New York University Journal of Law and Liberty , 12 (2), 333-351.
- Fish, S. (2018b). The Interpretive Poverty of Data. Balkinization . https://balkin.blogspot.com/2018/03/the-interpretive-poverty-ofdata.html
- Flower, L., & Hayes, J. R. (1981). A Cognitive Process Theory of Writing. College Composition and Communication , 32 (4), 365387. https://doi.org/10.2307/356600
- Gavin, M. (2020). Is there a text in my data? (Part 1): On counting words. Journal of Cultural Analytics . https://doi.org/10.22148/001c.11830
- Gibson, J. J. (1979). The Ecological Approach to Visual Perception . Houghton Mifflin.

## Scandinavian Studies in Language, 12(1), 2021 (1-29)

- Graham, S., Kiuhara, S. A., Harris, K. R., & Fishman, E. J. (2017). The Relationship among Strategic Writing Behavior, Writing Motivation, and Writing Performance with Young, Developing Writers. The Elementary School Journal , 118 (1), 82-104. https://doi.org/10.1086/693009
- Graham, S., McKeown, D., Kiuhara, S., & Harris, K. R. (2012). A metaanalysis of writing instruction for students in the elementary grades. Journal of Educational Psychology , 104 (4), 879-896. https://doi.org/10.1037/a0029185
- Graham, S., & Perin, D. (2007). A Meta-Analysis of Writing Instruction for Adolescent Students. Journal of Educational Psychology , 99 (3), 445-476. https://doi.org/10.1037/0022-0663.99.3.445
- Gray, B. (2015). Linguistic Variation in Research Articles. When discipline tells only part of the story . John Benjamins Publishing Company. https://doi.org/10.1075/scl.71
- Halliday, M. A. K. (1978). Language as Social Semiotic . Edward Arnold.
- Halliday, M. A. K. (2013). Halliday's Introduction to Functional Grammar (4th edition). Routledge.
- Hardy, J. A., & Römer, U. (2013). Revealing disciplinary variation in student writing: A multi-dimensional analysis of the Michigan Corpus of Upper-level Student Papers (MICUSP). Corpora , 8 (2), 183-207. https://doi.org/10.3366/cor.2013.0040
- Hirst, G. (2013). Computational Linguistics. In K. Allan (Ed.), The Oxford Handbook of The History of Linguistics (pp. 707-726). Oxford University Press.
- Hutchby, I. (2001). Technologies, Texts and Affordances. Sociology , 35 (2), 441-456. https://doi.org/10.1177/S0038038501000219
- Hyland, K. (2004). Disciplinary Discourses: Social Interactions in Academic Writing . University of Michigan Press.
- Jewitt, C. (2006). Technology, literacy and learning: A multimodal approach . Psychology Press.
- Jurafsky, D. (2002). Probabilistic Modeling in Psycholinguistics: Linguistic Performance and Production. In R. Bod, J. Hay, & S. Jannedy (Eds.), Probabilistic Linguistics (pp. 39-96). MIT Press.
- Juuhl, G. K. (2020). Text in and out of school - a dialogical rhetorical analysis. Nordic Journal of Literacy Research , 6 (1), Article 1. https://doi.org/10.23865/njlr.v6.1419

## Scandinavian Studies in Language, 12(1), 2021 (1-29)

- Kern, R. (2000). Literacy and language teaching . Oxford University Press.
- Kress, G. (1994). Learning to write (2. ed.). Routledge.
- Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication . Routledge.
- Kress, G., & van Leeuwen, T. (1996). Reading Images: The Grammar of Visual Design . Routledge.
- Kvistad, A. H., & Otnes, H. (2019). Mottakerinstansen i skoleskriving En studie av skriveoppgaver fra Normprosjektet. Nordic Journal of Literacy Research , 5 (2). https://doi.org/10.23865/njlr.v5.1644
- Kyle, K. (2021). Natural language processing for learner corpus research. International Journal of Learner Corpus Research , 7 (1), 1-16.
- Lang, S. (2018). Evolution of Instructor Response? Analysis of Five Years of Feedback to Students. Journal of Writing Analytics , 2 , 133. https://doi.org/10.37514/JWA-J.2018.2.1.02
- Lankshear, C., & Knobel, M. (2003). New literacies: Changing knowledge and classroom learning . Open University Press.
- Leech, G. (1992). Corpora and theories of linguistic performance. In J. Svartvik (Ed.), Directions in Corpus Linguistics. Proceedings of Nobel Symposium 82 Stockholm, 4-8 August 1991 (pp. 105-122). De Gruyter Mouton.
- Leijten, M., van Waes, L., & Horenbeeck, E. van. (2015). Analyzing writing process data: A linguistic perspective. In G. Cislaru (Ed.), Writing(s) at the Crossroads: The process-product interface (pp. 277-302). John Benjamins Publishing Company. https://doi.org/10.1075/z.194.14lei
- Manovich, L. (2001). The Language of New Media . MIT Press.
- Manovich, L. (2020). Cultural Analytics . MIT Press.
- McEnery, T., & Hardie, A. (2013). The History of Corpus Linguistics. In K. Allan (Ed.), The Oxford Handbook of The History of Linguistics (pp. 727-745). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199585847.013.0034
- McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2009). Linguistic Features of Writing Quality: Written Communication , 27 (1), 5786. https://doi.org/10.1177/0741088309351547

Scandinavian Studies in Language, 12(1), 2021 (1-29)

- Mohr, J. W., Wagner-Pacifici, & Breiger, R. L. (2015). Toward a computational hermeneutics. Big Data & Society , 2 (2), 1-8. https://doi.org/10.1177/2053951715613809
- Moretti, F. (2000). Conjectures on World Literature. New Left Review , 1 . Moretti, F. (2013). Distant reading . Verso.
- Moxley, J., Elliot, N., Eubanks, D., Vezzu, M., Elliot, S., & Allen, W. (2017). Writing Analytics: Conceptualization of a Multidisciplinary Field. Journal of Writing Analytics , 1 , v-xvii. https://doi.org/10.37514/jwa-j.2017.1.1.02
- Murphy, S., & Yancey, K. B. (2007). Construct and Consequence: Validity in Writing Assessment. In C. Bazerman (Ed.), Handbook of Research on Writing (pp. 448-473). Routledge.
- O'Keeffe, A., McCarthy, M., & Carter, R. (Eds.). (2007). Exploring teacher corpora. In From Corpus to Classroom: Language Use and Language Teaching (pp. 220-245). Cambridge University Press. https://doi.org/10.1017/CBO9780511497650.012
- Olinghouse, N. G., & Wilson, J. (2013). The relationship between vocabulary and writing quality in three genres. Reading and Writing , 26 (1), 45-65. https://doi.org/10.1007/s11145-012-9392-5
- Palermo, G. J. (2017). Transforming Text: Four Valences of a Digitial Humanities Informed Writing Analytics. Journal of Writing Analytics , 1 , 311-343. https://doi.org/10.37514/jwa-j.2017.1.1.11
- Parsons, S. A., Gallagher, M. A., Leggett, A. B., Ives, S. T., & Lague, M. (2020). An Analysis of 15 Journals' Literacy Content, 2007-2016. Journal of Literacy Research , 52 (3), 341-367. https://doi.org/10.1177/1086296X20939551
- Perelman, L. (2012). Construct validity, score, and time in holistically graded writing assessments: The case against automated essay scoring (AES). In C. Bazerman, C. Dean, J. Early, K. Lunsford, S. Null, P. Rogers, & A. Stansell (Eds.), International Advances in Writing Research: Cultures, Places, Measures (pp. 83-101). Fort Collins, Colorado: WAC Clearinghouse/Anderson, SC: Parlor Press. https://doi.org/10.37514/per-b.2012.0452.2.07
- Perelman, L. (2020). The BABEL generator and E-rater: 21st Century Writing Constructs and Automated Essay Scoring (AES). Journal of Writing Assessment , 13 (1), 1-9.

## Scandinavian Studies in Language, 12(1), 2021 (1-29)

- Reppen, R. (2010). Using Corpora in the Language Classroom . Cambridge University Press.
- Römer, U., Cortes, V., & Friginal, E. (2020). Advances in Corpus-Based Research on Academic Writing: Effects of Discipline, Register, and Writer Expertise (Vol. 95). John Benjamins Publishing Company. https://doi.org/10.1093/applin/amaa065
- Rousse-Malpat, A., Steinkrauss, R., & Verspoor, M. (2019). Structurebased or dynamic usage-based instruction: Long-term effects on (morpho)syntactic and lexical complexity in writing samples. Instructed Second Language Acquisition , 3 (2), 181-205. https://doi.org/10.1558/isla.38054
- Searle, J. R. (1969). Speech acts: An essay in the philosophy of language (Repr.). Cambridge University Press.
- Searle, J. R. (1995). The Construction of Social Reality . Free Press.
- Selwyn, N. (2015). Data entry: Towards the critical study of digital data and education. Learning, Media and Technology , 40 (1), 64-82. https://doi.org/10.1080/17439884.2014.921628
- Selwyn, N. (2019). What's the Problem with Learning Analytics? Journal of Learning Analytics , 6 (3), 11-19-11-19. https://doi.org/10.18608/jla.2019.63.3
- Shermis, M. D. (2014). State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration. Assessing Writing , 20 , 53-76. https://doi.org/10.1016/j.asw.2013.04.001
- Shermis, M. D., & Burstein, J. (Eds.). (2013). Handbook of Automated Essay Evaluation: Current Applications and New Directions . Taylor & Francis Group. https://doi.org/10.4324/9780203122761.ch1
- Siemens, G. (2013). Learning Analytics: The Emergence of a Discipline. American Behavioral Scientist , 57 (10), 1380-1400. https://doi.org/10.1177/0002764213498851
- Street, B. V. (1984). Literacy in theory and practice . University Press.
- Street, B. V. (1995). Social literacies: Critical approaches to literacy in development, ethnography and education. Longman.
- Stubbs, M. (2007). On texts, corpora and models of language. In M. Hoey, M. Mahlberg, M. Stubbs, & W. Teubert (Eds.), Text,

## Scandinavian Studies in Language, 12(1), 2021 (1-29)

Discourse and Corpora: Theory and Analysis (pp. 127-161). Continuum.

- Swales, J. M. (1990). Genre analysis: English in academic and research settings . Cambridge University Press.
- ten Peze, A., Janssen, T., Rijlaarsdam, G., & van Weijen, D. (2021). Writing creative and argumentative texts: What's the difference? Exploring how task type affects students' writing behaviour and performance. L1 - Educational Studies in Language and Literature , 21 , 1-38. https://doi.org/10.17239/L1ESLL2021.21.01.11
- The New London Group. (1996). A Pedagogy of Multiliteracies: Designing Social Futures. Harvard Educational Review , 66 (1), 60-92. https://doi.org/10.4324/9780203979402-6
- Togeby, O. (2010). Handling, tekstualisering og tekst. Scandinavian Studies in Language , 1 (1), 67-92. https://doi.org/10.7146/sss.v1i1.3790
- Togeby, O. (2015). Den dobbelte genreforventning: Genreforhold og sproglige valg i elevtekster. In E. Krogh, T. S. Christensen & K. S. Jakobsen (Eds.), Elevskrivere i gymnasiefag (pp. 267-292). Syddansk Universitetsforlag.
- Tracy-Ventura, N., & Paquot, M. (2020). The Routledge Handbook of Second Language Acquisition and Corpora . Routledge. https://doi.org/10.4324/9781351137904
- Troelsen, S. (2018). En invitation man ikke kan afslå - analyse af afgangsprøven i skriftlig fremstilling med særligt fokus på skriveordren. Nordic Journal of Literacy Research , 4 (1), 142-166. https://doi.org/10.23865/njlr.v4.1267
- Troia, G. A., Shen, M., & Brandon, D. L. (2019). Multidimensional Levels of Language Writing Measures in Grades Four to Six. Written Communication , 36 (2), 231-266. https://doi.org/10.1177/0741088318819473
- Tuldava, J. (1998). Probleme und Methoden der quantitativ-systemischen Lexicologie . Wissenschaftlicher Verlag.
- van Es, K., Wieringa, M., & Schäfer, M. T. (2018). Tool Criticism: From Digital Methods to Digital Methodology. In E. Reyes, M. Bernstein, G. Ruffo, & I. Saleh (Eds.), WS.2 2018: Proceedings of the 2nd International Conference on Web Studies (pp. 24-27).

Morten Tannert

## Scandinavian Studies in Language, 12(1), 2021 (1-29)

Association for Computing Machinery. https://doi.org/10.1145/3240431.3240436 (2),

Wise, A. F., & Shaffer, D. W. (2015). Why Theory Matters More than Ever in the Age of Big Data. Journal of Learning Analytics , 2 5-13. https://doi.org/10.18608/jla.2015.22.2