# The Potential Impact of EFL Textbook Language on Learner English: A Triangulated Corpus Study

Elen Le Foll

<figure><img src="image_1.png" /></figure>

## To cite this version:

Elen Le Foll. The Potential Impact of EFL Textbook Language on Learner English: A Triangulated Corpus Study. Demystifying Corpus Linguistics for English Language Teaching, Palgrave MacMillan, pp.259-287, 2023, 978-3-031-11220-1. 10.1007/978-3-031-11220-1_13. hal-04070447

HAL Id: hal-04070447

https://hal.science/hal-04070447

Submitted on 15 Apr 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
This is a **pre-print version** of a paper now published as:

Le Foll, Elen. 2023. The Potential Impact of EFL Textbook Language on Learner English: A Triangulated Corpus Study. In Kieran Harrington & Patricia Ronan (eds.), *Demystifying Corpus Linguistics for English Language Teaching*, 259–287. Cham: Palgrave MacMillan.
https://doi.org/10.1007/978-3-031-11220-1_13.

This is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: https://doi.org/10.1007/978-3-031-11220-1_13. Users may only view, print, copy, download and text- and data-mine the content of this pre-print for the purposes of academic research. The content may not be (re-)published verbatim in whole or in part or used for commercial purposes. Users must ensure that the author's moral rights as well as any third parties' rights to the content or parts of the content are not compromised

# The Potential Impact of EFL Textbook Language on Learner English: A Triangulated Corpus Study*

Elen Le Foll, Osnabrück University

elefoll@uni-koeln.de

ORCID.org/0000-0002-5839-8010

This chapter investigates how periphrastic causative constructions (e.g., *What makes you think that?*) are represented in secondary school EFL textbooks and how textbook representations may influence the output of EFL learners. First, the explicit descriptions of causative constructions in textbooks are examined. Second, the syntactic and phraseological features of the causative constructions featured in a corpus of 42 textbooks are analysed. Third, the results are compared to previous corpus-based research on how EFL, ESL and ENL students use causatives. The comparisons lend support to Gilquin's (2016) input-dependent L2 acquisition hypothesis which suggests that differences between EFL and ESL learner corpus data may be (partially) attributable to EFL learners' minimal exposure to natural language input. For instance, by far the most frequent construction in the examined textbooks, [X MAKE Y V<sub>inf</sub>], was previously shown to be vastly overused by EFL learners (e.g., Wong 1983; Gilquin 2016), whilst constructions that are underused by learners as compared to ENL data (e.g., Gilquin 2012; 2016) are very rare in the textbook data. Low frequencies of specific constructions in textbooks also correlate with EFL learners producing high rates of syntactic errors when attempting these constructions.

<footnote>* An earlier version of this manuscript was entitled: *Textbooks and Corpus Linguistics: the case of causative constructions.*</footnote>
# 1. Introduction

In instructed foreign language learning contexts and, in particular, at lower secondary school level, a considerable proportion of learners’ language input comes from their textbooks (Volkmann 2010: 235). Given their relevance, it will come as no surprise that there is now a growing body of corpus-linguistic studies analysing what will be referred to here as “Textbook English”. Most studies to date have compared Textbook English to that of naturally-occurring or “authentic” English as a Native Language (ENL) and many have reported major discrepancies in the use of individual lexico-grammatical features such as future constructions (e.g., Mindt 1987), the progressive (e.g., Römer 2005), and the phraseological patterns of high-frequency verbs (e.g., Le Foll 2022a). Far fewer studies, however, have explored potential correlations between Textbook English and the language produced by learners of English as a Foreign Language (EFL) as the recipients of these textbooks. Exceptions include triangulated investigations of *if*-conditionals (e.g., Gabrielatos 2013; Möller 2020; Winter & Le Foll 2022), which have, on the one hand, shown that the representations of *if*-conditionals in EFL textbooks are far removed from naturalistic ENL use and, on the other, that EFL learners are influenced by the types of *if*-sentences featured in such pedagogical materials, even at higher levels of English proficiency.

Since they entail the comparison of three corpora, such triangulated, Textbook English vs. ENL vs. Learner English, studies are rarely conducted. The present study shows how the results of existing EFL learner vs. ENL corpus research may be drawn upon to explore whether and, if so, how textbook language input may influence learners’ syntactic and lexical choices. As an example of this influence, the current study investigates the potential impact of Textbook English on EFL learners’ use of causative constructions. Causative constructions were chosen as an example of a construction that straddles textbooks’ traditional binary distinctions between vocabulary and grammar and because they have been the focus of a number of learner corpus studies that have pointed to persistent issues in learners’ use of this construction (e.g., Wong 1983; Altenberg & Granger 2001; Liu & Shaw 2001; Gilquin 2012; 2016).
The following section introduces the theoretical background and summarises insights from learner English corpus studies on causatives. Section 3 presents the textbook corpus data queried as part of this investigation, the procedure adopted to process the data, and the method employed for the phraseological analysis. Section 4 begins with the results of the qualitative analysis of the descriptions of causative constructions featured in the textbook grammar sections and continues with the corpus-based analysis of the causatives implicitly featured within the texts of the same textbooks. The representations of causative constructions in textbooks are then compared to the results of previous corpus-based learner English studies with the aim of teasing out the potential impact of Textbook English on EFL learners' language production, before showing how corpora can be used to supplement textbook materials.

## 2. Theoretical background

### 2.1 Causatives in the Construction Grammar framework

The present study focuses on EFL textbooks' representations of periphrastic causative constructions with non-finite verb complements such as:

(1) Has this lesson *made you consider* changing the way you use your mobile phone in public? <TEC: Achievers B2>†

These constructions consist of a causative verb and a non-finite verb complement clause. The causative verb (e.g., in example (1), *make*) is governed by the causer (*this lesson*) and controls the complement clause which defines both the causee (*you*) and the effect exerted by the causer (*consider changing the way you use your mobile phone in public*). The most frequent causative verbs are MAKE (1), CAUSE (2), GET (3) and HAVE (4):

(2) What situations *cause people to make* faces like that? <TEC: Green Line New 5>

(3) Josh *got me to try* bungee-jumping. <TEC: Piece of Cake 3e>

(4) She *had her old phone repaired*, even though she expected to get a new one.
<TEC: Solutions intermediate>

Causative constructions in English have been extensively described within a number of theoretical frameworks (2016: 115). The present study adopts a Construction Grammar (e.g.,

<footnote>† In this and the following examples, the code between angle brackets denotes the source of the example, i.e. the abbreviation of the corpus followed by the relevant subcorpus (e.g., textbook volume, learner L1 language) or document ID. Unless otherwise stated, emphasis was added.</footnote>
Goldberg 1995; 2006) approach. Within this framework, constructions are, at the most basic level, understood as “conventionalized pairings of form and function” (Goldberg 2006: 3) and are seen as the basic units of language. Traditional English grammars have usually presented the different types of English causative constructions as more or less synonymous (see Gilquin 2010: 98–99 for overview). However, within a Cognitive Linguistics (e.g., Langacker 2008) and, more specifically, Construction Grammar framework, the principle of “one form, one meaning” prevails. In other words, “[i]f two constructions are syntactically distinct, they must be semantically or pragmatically distinct”, too (Goldberg 1995: 67; see also Croft’s 2019: 111– 112 “Principle of Contrast”). By combining the analysis of corpus and elicitation data, Gilquin (2010: Ch. 5) demonstrates that this principle of non-synonymy applies to periphrastic causative constructions. Though many syntactic and semantic features are distributed in significantly different ways across different causative constructions, the factor that best predicts the choice of one causative construction over another is a lexical one: namely the verb used to describe the EFFECT in the non-finite complement clause (Gilquin 2010: 138–139), e.g., DO in (5).

(5) yeah yeah it is a long operation [...]
of course he won't be able to work for weeks and weeks
no this is it that's why [he] hasn't *had it done* he *could have had it done*
I thought he did
he *had it done* once but he's got more joints now <BNC1994: SQS3>

Construction Grammar assumes that speakers' language knowledge consists of "a network of learned pairings of form and function, or constructions" (Goldberg 2005: 17). These range from morphemes to individual words, to collocations and verb-argument structures. Whilst the idea that language can be broken down to constructions that span right across the traditional grammar-lexicon divide is no longer new, it remains controversial in many a linguistic circle (Langacker 2008: 22). In foreign language pedagogy, however, approaches advocating remarkably compatible ideas go back a long way (e.g., Hausmann 1984). Yet such approaches have yet to be reflected in most of the EFL textbooks currently in use in secondary schools, which still tend to apply a strict (but often largely arbitrary) distinction between grammar units on the one hand and vocabulary on the other.

## 2.2 Causatives in English Language Teaching and Learner English
Periphrastic causative constructions are both very frequent and of high communicative value; however, they form complex syntactic patterns which often prove problematic for learners of English. Gilquin (2012) shows with data from the International Corpus of Learner English (ICLE; Granger et al. 2009) that even proficient learners of English frequently confuse the different verb complements of causative verbs (e.g., using a *to*-infinitive instead of the bare infinitive with MAKE as in (6)) or struggle with the placement of the causee (e.g., (7)).

(6) It is good practice to use English every day. That *makes us not to forget* English grammar or words. <ICLE: Japanese L1>

(7) Compensation orders *could make feel parents* more responsible for their sons and daughters. <ICLE: Italian L1>

In addition to syntactic issues, EFL learners tend to significantly over- or underuse certain types of causative constructions as compared to ENL speakers (Wong 1983; Altenberg & Granger 2001; Liu & Shaw 2001; Gilquin 2012; 2016). These same studies have also shown that learners frequently make infelicitous or non-idiomatic lexical choices when producing causative constructions. For example, Gilquin (2016) compared the lexical choices of native English (ENL), English as a Second Language (ESL) and English as a Foreign Language (EFL) speakers for the non-finite verb slot of causative constructions with MAKE. The ESL vs. EFL learner comparison is particularly interesting in the context of the present study because, whilst most EFL users will have, at least at first, learnt English as a school subject with few additional language inputs other than their textbooks, ESL users have usually completed large parts of their education in English and/or live in countries where English is an official language (e.g., India, Nigeria, Singapore) and are therefore exposed to much more non-textbook materials than most EFL learners.

Gilquin (2012) identifies three possible causes for learners’ unidiomatic use of certain periphrastic causative constructions: a) lack of register awareness, b) negative L1 transfer, and c) the inadequacy of teaching materials. The influence of this third potential factor (also identified by Altenberg & Granger [2001: 184] as a potential source of learners’ misuse of MAKE causatives) was further explored in Gilquin (2016), whose results suggest that different types of language input may indeed be partially responsible for the differences observed between EFL and ESL learners’ use of causatives. Gilquin’s (2016) results support her input-dependent L2 acquisition hypothesis which states that:
considering that (i) EFL learners learn English mainly in an instructional setting and receive relatively little exposure to the target language, and (ii) ESL learners, in addition, also acquire English in a natural setting and thus receive more exposure to the target language, it can be expected that ESL learners will have been exposed to more (authentic) instances of a given construction than EFL learners and will thus have better integrated its schema, resulting in a better, more native-like command of the construction. (Gilquin 2016: 119)

This hypothesis, which raises the question of the (in)adequacy of teaching materials in EFL instructional settings, is at the heart of the present study. It draws on a large corpus of EFL textbooks to explore the distributional patterns of the syntactic, semantic, and phraseological characteristics of the causative constructions to which learners are frequently exposed to via their textbooks. Moreover, it explores whether textbooks provide learners with adequate input that will help them produce these constructions idiomatically in context.

# 3. Data and methodology

## 3.1 Corpus data

The Textbook English Corpus (TEC; cf. Le Foll 2021a; 2022a; 2022b) aims to capture the language input that lower secondary school pupils in France, Germany and Spain obtain from their English textbooks. The corpus consists of nine series of best-selling EFL textbooks from eight publishers. The textbooks have been manually annotated to enable fine-grained comparisons between learner levels, learner L1 and text registers (see Appendix I). Seven register subcorpora of the TEC were included in the present analysis: Informative, Instructional, Narrative, Other, Personal, Poetry and Spoken (see Appendix II). Together, these subcorpora comprise 1,8 million words (hereafter referred to collectively as the TEC-T).

## 3.2 Data extraction and processing

This section describes how the causative constructions of the TEC-T were extracted for the second part of the analysis. In the context of a pedagogically-oriented study, it made sense to focus on the most frequent causative verbs: MAKE, CAUSE and GET.‡ The first step was thus to locate and analyse all the causative constructions of the TEC-T with these three verbs. To this

<footnote>‡ The extraction of causative HAVE constructions was not attempted because these are relatively rare and their extraction would have required a lot of manual work. Indeed, Gilquin (2010: 37) reported that only ca. 0.35% of all occurrences of HAVE are instances of causative constructions.</footnote>
end, the corpus was uploaded onto the web-based corpus analysis tool Sketch Engine (Kilgarriff et al. 2014). Sketch Engine provides a layer of automatic linguistic analysis which makes it possible to query the data on the basis of part-of-speech information (e.g., whether the word *make* is a noun, a verb in infinitive form, or present tense) and lemmas (e.g., *make*, *makes*, *making* and *made* are all forms of the lemma *MAKE*). Due to the high frequency of the verbs *MAKE* and *GET*, the extraction of these constructions was divided into two phases: search queries were used to automatically extract all possible instances of the constructions and these were then manually sorted to narrow the results down to actual occurrences of causative constructions. Thus, for the first phase, it was necessary to find queries that would be precise enough to considerably reduce the second time-consuming phrase but, at the same time, would have the highest possible recall rate so as not to miss any relevant constructions. After some experimentation, the following queries were chosen for the automatic extraction phase:

(8) [lemma="make" & tag="V.*"] [tag!="SENT|,"]{0,5} [tag="V.*"]

(9) [lemma="get"] [tag!="IN|V.N|V.G" & word!="into|to"] [tag="N.*|PP.?"]{1,5}
[tag="V.*"]{1,7}

(10) [lemma="cause" & tag="V.*"]

The queries are formulated in Corpus Query Language (CQL). The first, (8), returns all occurrences of the verb MAKE followed by another verb form (operationalised by [tag="V.*"]) with up to five intervening tokens in between to allow for longer nominal phrases as direct objects. A token is a word-like unit which, however, can also include punctuation marks which is why query (8) specifies that the five intervening tokens may be of any kind except end-of-sentence punctuation or commas. The second query for GET causatives, (9), is more complex because it attempts to exclude a number of other frequent GET constructions such as passives (e.g., *He got fined*) and constructions of intransitive motion (e.g., *How did she get into this?*). As CAUSE is a relatively infrequent verb, its CQL query, (10), is much simpler: it simply retrieves occurrences of CAUSE as a verb and, since they are only 105 in the TEC-T, these were then manually disambiguated.

All causative constructions were subsequently manually annotated for their intermediate-level construction following Gilquin's (2010: 222) active-passive classification (see Table 1 for subtypes of MAKE constructions), as well as for the lemma of the verb occurring in the non-finite verb slot (e.g., APPEAR, HEAR, KEEP and KNOW in the examples of Table 1).
<table><caption>**Table 1:** Intermediate-level causative constructions with MAKE</caption><thead><tr><td>Intermediate-level construction</td><td>Example</td></tr></thead><tbody><tr><td>[MAKE<sub>act</sub> V<sub>act</sub>]</td><td>This is often done deliberately to <i>make</i> a product <i>appear</i> more popular. &lt;TEC: Achievers B1+&gt;</td></tr><tr><td>[MAKE<sub>act</sub> V<sub>pas</sub>]</td><td>How can you <i>make</i> yourself <i>heard</i>? &lt;TEC: Piece of Cake 4<sup>e</sup>&gt;</td></tr><tr><td>[MAKE<sub>pas</sub> V<sub>act</sub>]</td><td>It was <i>made to keep</i> the devil away. &lt;TEC: Join the Team 5<sup>e</sup>&gt;</td></tr><tr><td>[MAKE<sub>pas</sub> V<sub>pas</sub>]</td><td>There is nothing secret that will not be <i>made known</i>. &lt;BNC1994: A18&gt;</td></tr></tbody></table>

### 3.3 Collostructional analysis

Collostructional analysis explores the interactions between grammatical constructions and the content words (lexemes) associated with them (cf. Stefanowitsch & Gries 2003). As such, the method also follows a construction-based approach to language since it assumes that grammar, just like the lexicon, consists of pairings of linguistic forms with linguistic meaning. Collostructional analysis takes a construction as a starting point and measures which lexemes are attracted to or repelled by the construction. To do so, it evaluates whether lexemes occur more or less frequently than their expected frequency on the basis of the total lexeme and construction counts in the corpus under study. The lexemes associated with a particular construction are referred to as its collexemes (Stefanowitsch & Gries 2003: 215).

The analysis was carried out using the Coll analysis 3.2a package for R (Gries 2007). As recommended by Stefanowitsch and Gries (2003: 218), the association measure adopted here is Fisher's exact test because it does not rely on or make any distributional assumptions (which are frequently not met when working with corpus data), nor does it impose specific sample size demands. Nonetheless, like all phraseological measures, collostructional analysis requires relatively large sample sizes to yield significant results. The causative constructions data extracted from the TEC-T only provided enough data to perform a collexeme analysis on the most frequent causative construction: [X MAKE Y V<sub>inf</sub>].

## 4. Causatives in EFL textbooks

### 4.1 Causatives in textbook grammars

The term *causative* is mentioned in five out of the nine textbook series of the TEC (see Appendix I). The term is found in the following volumes: *Piece of Cake* 4<sup>e</sup>, *New Bridges* 2<sup>e</sup>, *New Missions* 2<sup>e</sup>, *English in Mind* 3 and *Solutions B2 Intermediate Plus*. The first three represent one volume from each of the three French textbook series of the TEC. This is because,
in France, causative constructions are expected to be explicitly taught as part of the lower secondary school EFL curriculum.

*New Bridges 2*<sup>e</sup> (see extract (11) from p. 196; emphasis original) focuses on potential syntactic difficulties when translating the French construction *faire faire* and, at the end of the explanation, points to a semantic difference between *MAKE* and *HAVE*, arguing that *MAKE* implies more of a constraint than *HAVE*.

(11) « faire faire »

◎ MAKE *ou HAVE + base verbale*

*The police made the terrorist talk. [...]*

*Juliet had us talk all night. [...]*

→ MAKE exprime plus la contrainte que HAVE.

◎ HAVE + *participle passé*

*We’ve had the lock changed. [...]*

→ On ne nous dit pas qui a changé la serrure mais seulement que la serrure a été changée. Le participe passé a un sens passive. MAKE est impossible dans ce cas.

◎ Traduction de *FAIRE FAIRE*

Il faut se demander si l'énoncé a un sens *passif* ou *actif*.

■ Sens passif : HAVE + participe passé [...]

» She's had some food delivered.

■ Sens actif (quelqu'un accomplit une action à la demande de quelqu'un d'autre):

MAKE *ou HAVE + base verbale* [...]

» She made / had them go out of the room.

In its grammar point on causative constructions, *New Missions 2*<sup>e</sup> also takes the translation of *faire faire* as its starting point. It, too, focuses on the differences in the syntactic patterns of the passive and active forms. In addition, it draws a semantic continuum across MAKE, HAVE and LET from the most constraint to the least (where “authorisation” is considered to be the least constrained – though it may be argued that authorisation does not necessarily imply causation) (see Table 2). All of its example sentences are drawn from the semantic field of law.
<table><caption>**Table 2:** Extract from *New Missions* 2<sup>e</sup>, p. 208 (emphasis original)</caption><thead><tr><th>verbe</th><th>construction<br>(base verbale)</th><th>degré de pression</th></tr></thead><tbody><tr><td><i>make</i></td><td><i>someone do something</i></td><td><i>Her lawyer made her admit the crime.</i><br>Forte contrainte</td></tr><tr><td><i>have</i></td><td></td><td><i>He had the prisoner pay for his crime.</i><br>Pression moins forte</td></tr><tr><td><i>let</i></td><td></td><td><i>He let his lawyer talk.</i><br>Autorisation</td></tr></tbody></table>

*New Missions* 2<sup>e</sup> (p. 208) also lists *allow*, *ask*, *cause*, *encourage/incite/urge*, *force/oblige*, *get*, *order* and *persuade* as causative verbs. However, these are not further explained or illustrated with any example sentences.

*Piece of Cake* 4<sup>e</sup> (p. 166) features a relatively detailed description of causative constructions with the verbs *HAVE*, *MAKE* and *LET*. Aside from analysing their syntactic structures, it also contrasts a number of semantic aspects. In particular, the authors claim that *HAVE* causatives are used to delegate actions, whilst *MAKE* causatives imply that a (forced) action or an emotion is provoked.

The two non-French textbooks that explicitly mention causative constructions only do so fleetingly. *English in Mind* 3 (pp. 49 and 53) focuses on *HAVE* constructions in the sense of *HAVE* (*something*) *done* and its exercises on the subject suggest that the semantics of the construction are limited to services provided to individuals (the highlighted prototypical example being: *having your hair done*), whilst *Solutions B2 Intermediate Plus* (p. 130) simply lists *GET* and *HAVE* as verbs entering causative constructions in the following pattern *verb + object + past participle* without providing any further information.

This section has provided an overview of how causative constructions are explicitly presented in the grammar sections of five EFL textbook series. The following sections investigate the nature of the *MAKE*, *CAUSE* and *GET* causatives that learners are exposed to implicitly when learning English with the textbooks of the TEC. The results are then compared to the results of existing EFL learner corpus research in Section 5.
## 4.2 Syntactic analysis of causative constructions in the TEC-T

Table 3 displays the raw and relative frequencies (per million words, hereafter pmw) of the causative constructions extracted from the TEC-T with the combination of the automatic corpus queries and manual sorting (see 3.2). A cursory glance suffices to conclude that one construction is featured much more prominently than any other: [X MAKE Y V<sub>inf</sub>], e.g., (1). It accounts for 97% of MAKE causative constructions in the TEC-T. Whilst some of the textbooks focus their grammar sections on causatives on the syntactic difficulty involved in the use the causative verb MAKE in the passive (see 4.1), the [X BE made V<sub>to-inf</sub>] construction is largely absent from the actual language input delivered by the textbooks: it is featured just once in two textbook series (*Access* and *Join the team*) and twice in *Green Line New*. No single occurrence of [MAKE<sub>pas</sub> V<sub>pas</sub>] was observed across the entire TEC-T.

<table><caption>**Table 3:** Causative constructions in the TEC-T</caption><thead><tr><td>Construction</td><td>Raw frequency in the TEC-T</td><td>Relative frequency in the TEC-T (pmw)</td></tr></thead><tbody><tr><td>[X CAUSE Y V<sub>to-inf</sub>]</td><td>8</td><td>5.98</td></tr><tr><td>[X GET Y V<sub>pp</sub>]</td><td>13</td><td>9.71</td></tr><tr><td>[X GET Y V<sub>prep</sub>]</td><td>2</td><td>1.49</td></tr><tr><td>[X MAKE Y V<sub>inf</sub>]</td><td>289</td><td>215.88</td></tr><tr><td>[X MAKE Y V<sub>pp</sub>]</td><td>5</td><td>3.73</td></tr><tr><td>[X BE made V<sub>to-inf</sub>]</td><td>4</td><td>2.99</td></tr></tbody></table>

## 4.3 Collostructional analysis of the [X MAKE Y V<sub>inf</sub>] construction in the TEC-T

Table 4 lists the collexemes of the [X MAKE Y V<sub>inf</sub>] construction with the strongest collostructional strength (see Appendix III for full results). The collostructional strength column indicates whether the co-occurrence frequency of a verb collexeme in the construction is significantly different to what we would expect under the assumption of no association given the construction's and collexeme's overall frequencies within the TEC-T (these correspond to the expected frequencies). The higher the value, the more strongly this verb is associated with the construction. The verb most strongly associated to the [X MAKE Y V<sub>inf</sub>] construction in the TEC-T is, by far, FEEL, e.g.:

(12) Hobbies are excellent ways to *make you feel* happy and they can also be a way of making friends. <TEC: Hi There 5e>
Other strongly associated verbs include more verbs of emotion (CRY, LAUGH) and verbs of cognition (THINK, WONDER, PROMISE, CONSIDER, UNDERSTAND), e.g.:

(13) She's the funniest person in our family – she can really *make* you *laugh*. <TEC: English in Mind 1>

(14) Do you think the people in the photo spend a lot of time at the gym? What *makes* you *think* so? <TEC: Solutions B1 Intermediate>

<table><caption>**Table 4:** Most significant results from the collexeme analysis of [X MAKE Y V<sub>inf</sub>] in the TEC-T (for full results see Appendix III)</caption><thead><tr><th>Collexeme lemma</th><th>Lemma frequency in the TEC-T</th><th>Observed frequency in [X MAKE Y V<sub>inf</sub>]</th><th>Expected frequency</th><th>Relation</th><th>Collostructional strength</th></tr></thead><tbody><tr><td>FEEL</td><td>1,412</td><td>65</td><td>1.27</td><td>attraction</td><td>88.74</td></tr><tr><td>LAUGH</td><td>252</td><td>14</td><td>0.23</td><td>attraction</td><td>20.36</td></tr><tr><td>SOUND</td><td>344</td><td>10</td><td>0.31</td><td>attraction</td><td>11.90</td></tr><tr><td>THINK</td><td>5,432</td><td>25</td><td>4.88</td><td>attraction</td><td>10.34</td></tr><tr><td>PROMISE</td><td>87</td><td>4</td><td>0.08</td><td>attraction</td><td>5.87</td></tr><tr><td>WONDER</td><td>133</td><td>4</td><td>0.12</td><td>attraction</td><td>5.14</td></tr><tr><td>SEEM</td><td>301</td><td>5</td><td>0.27</td><td>attraction</td><td>5.04</td></tr><tr><td>SNEEZE</td><td>14</td><td>2</td><td>0.01</td><td>attraction</td><td>4.14</td></tr><tr><td>CONSIDER</td><td>122</td><td>3</td><td>0.11</td><td>attraction</td><td>3.71</td></tr><tr><td>CRY</td><td>139</td><td>3</td><td>0.12</td><td>attraction</td><td>3.54</td></tr><tr><td>UNDERSTAND</td><td>654</td><td>5</td><td>0.58</td><td>attraction</td><td>3.46</td></tr><tr><td>WANT</td><td>3,280</td><td>10</td><td>2.94</td><td>attraction</td><td>3.05</td></tr><tr><td>DO</td><td>19,247</td><td>6</td><td>17.30</td><td>repulsion</td><td>-2.88</td></tr><tr><td>BE</td><td>73,499</td><td>1</td><td>66.06</td><td>repulsion</td><td>-30.80</td></tr></tbody></table>
**5. The potential impact of EFL textbooks on EFL writing**

In this section, the relative frequencies of causative constructions in the TEC-T are compared to the relative frequencies found in corpora of EFL and ESL learner essay writing.

**5.1 EFL learners' choice of causative constructions**

Gilquin (2016: 127) reports that the relative frequency of the [X MAKE Y V<sub>inf</sub>] construction is above 400 pmw in advanced EFL speakers (cf. Gilquin 2012: 8) and around 125 pmw in advanced ESL speakers. At 216 pmw, the relative frequency of this particular causative construction in the TEC-T is more than four times higher than in Gilquin's (2012: 8) reference ENL corpus (50.77 pmw). It is also considerably higher than in her ESL corpus data.

Only 13 instances of the [X GET Y V<sub>pp</sub>] construction were found in the TEC-T. Interestingly, the construction's relative frequency in the TEC-T (9.71 pmw) is noticeably closer to that found in Gilquin's (2012: 8) EFL data (7.73 pmw in ICLE) than in the reference ENL data (3.80 pmw). The [X GET Y V<sub>prp</sub>] construction has a similarly low frequency across all corpora. By contrast, Gilquin (2012) reported a significant underuse of the [X GET Y V<sub>to-inf</sub>] construction by EFL learners and, indeed, not a single occurrence of this construction is found in the textbooks under study.

The second most frequently used causative construction in Gilquin's (2016) ENL data is [X CAUSE Y V<sub>to-inf</sub>]. This construction is considerably less frequently in the TEC-T than in the ENL data. This difference may well be due to text register differences and would warrant further analysis with a comparable reference corpus. At the same time, however, it is worth noting that, according to Gilquin (2016), the [X CAUSE Y V<sub>to-inf</sub>] construction is the only causative construction which ESL learners significantly overuse compared to ENL writers. By contrast, EFL learners use it to approximately the same extent as ENL writers (cf. Gilquin 2012: 48). The verb CAUSE is not infrequent in the TEC-T, but only 8% of occurrences are in periphrastic constructions, compared to 44% in British ENL data (Gilquin 2010: 37).

Although the learners of the learner corpus studies cited above presumably did not learn with exactly the textbooks of the TEC-T, they can be assumed to have been exposed to very similar pedagogical materials; thus, these results suggest that how frequently a construction is presented in typical textbook materials influences learners' choices in their own language production (see also Winter & Le Foll 2022). Indeed, by far the most frequent causative
construction in the TEC-T, [X MAKE Y V<sub>inf</sub>], is seemingly overused by EFL learners, whilst
constructions that are underused either have a relative frequency that is considerably lower in
the TEC-T than in ENL writing, or are entirely absent from the 43 textbooks of the TEC.

**5.2 Unidiomatic syntax in EFL writing**

Not only does textbook-based language input appear to influence EFL learners’ choice of
construction, the results also suggest that low exposure from textbooks corresponds to an
increase in EFL users’ production of unidiomatic syntax. Whereas periphrastic constructions
with CAUSE occur more than 30 times pmw in ENL and EFL writing (Gilquin 2012: 48), in 4.2
we saw that they are very rare in the TEC-T. This finding sheds new light on Gilquin’s (2012:
50) conclusion that the causative constructions which trigger the most syntactic errors in EFL
writing are those involving the verb CAUSE: nearly a quarter of CAUSE constructions in ICLE
are reported to feature verb- and/or causee-related syntactic errors such as those illustrated in
(17) and (18) (taken from Gilquin 2012: 49–50).

(15) The ease payment of credit card *cause the students buying* too much and too quickly.
<ICLE: Chinese L1>

(16) *It causes to lose* their good emotions such as love, mercy and the other good ones.
<ICLE: Turkish L1>

Furthermore, Gilquin (2016: 127–128) demonstrates that ESL speakers produce CAUSE
constructions which follow the prototypical syntactic patterns used by ENL speakers
significantly more frequently than EFL learners. Gilquin interprets this finding as a
confirmation of the input-dependent L2 acquisition hypothesis because ESL speakers are likely
to have much richer, extra-curricular language input than EFL students. The results of the
present study support this hypothesis by pointing to the extremely low frequency of CAUSE
constructions in school EFL textbooks. Moreover, CAUSE is not featured in the explicit
treatment of causative constructions in the nine textbook series investigated (except for *New
Missions* 2e where it is mentioned once, but with no example sentence or additional
explanation, see 4.1).

Inversely, the results of the present corpus-based analysis of Textbook English suggest that
high exposure through textbook input may have a positive impact on learners’ production.
Active [X MAKE Y V<sub>inf</sub>] constructions have been shown to be considerably overused by EFL
learners (Gilquin 2012, 2016) and, indeed, this study shows (see 4.3) that they are, by far, the
most frequent causative constructions in the nine textbook series under study. In addition, Gilquin’s (2016: 128–137) syntactic error analysis in ESL and EFL essays reveals that, with [X MAKE Y V<sub>inf</sub>] constructions, EFL learners generally fare better than ESL students. Together, these findings suggest that EFL learners’ generally good assimilation of the syntactic peculiarity of the [X MAKE Y V<sub>inf</sub>] construction – which is unusual in that it takes a bare infinitive – may be, at least partially, the result of high exposure to this construction in EFL teaching materials.

The passive [X BE made V<sub>to-inf</sub>] construction, on the other hand, was shown to be largely absent from the TEC-T. If the syntactically unusual active [X MAKE Y V<sub>inf</sub>] construction is featured so prominently in textbooks, but the passive construction is more or less ignored, this is likely to have an impact on EFL learners’ idiomaticity when they do attempt the passive construction. Indeed, Gilquin (2016: 131) reports that whilst ESL students, who are likely exposed to more naturalistic language input, use native-like syntax in 100% of attested cases, some EFL learners, on the other hand, seemingly overgeneralise the rule from the active construction and thus produce sentences such (17) (from Gilquin 2016: 133).

(17) Czech students majoring in languages, for example, are often *made believe* that grammar is the most important part of language [...]. <ICLE: Czech L1>

Gilquin (2016: 136) hypothesises that MAKE, often perceived as the most prototypical causative verb (see also Altenberg 2002: 99), may “serve as a pathbreaking verb in instruction, that is, a verb that is used to introduce the general characteristics of the construction, before the construction is extended to other verbs.” Whilst this pedagogical strategy appears to bear fruit for the active [X make Y V<sub>inf</sub>] construction, it may be misleading some learners into assuming that all causative constructions follow this syntactic pattern with a bare infinitive. The present results thus underline the inseparability of grammar and vocabulary in language teaching and learning: each causative construction needs to be taught as a lexico-grammatical unit with its own syntactic patterns, semantics, and, as will be discussed in the following section, frequent lexical associations.

## 5.3 EFL learners’ use of the [X MAKE Y V<sub>inf</sub>] construction

The collexeme analysis in 4.3 shows that [X MAKE Y FEEL] is by far the most salient instantiation of causative constructions in the TEC-T. Interestingly, both Liu & Shaw (2001) and Gilquin (2012; 2016) report that FEEL is used significantly more frequently by EFL than
ENL users. In addition, BE and BECOME are also overused in this construction by learners (Liu & Shaw 2001: 179, Gilquin 2016: 139). In both cases, however, such an overuse of BE and BECOME is likely to be the result of learners using a verb complement where an adjective complement would have sufficed, see example (20) from Gilquin (2012: 14), and, as such, we would not expect textbooks to feature such infelicitous constructions.

(18) That will *make it be* more popular. <ICLE: Tswana L1>

Indeed, the results in Table 4 show that BE is the verb lemma most strongly repelled by the [X MAKE Y V<sub>inf</sub>] construction. Thus, unlike the use of FEEL, learners' overuse of BE and BECOME in this construction does not appear to be textbook-induced. This finding also supports the input-dependent L2 acquisition hypothesis as Gilquin (2016: 139–140) observes an overuse of BE and BECOME for both ESL and EFL learners, whereas the overuse of FEEL is seemingly exclusive to EFL learners who can be assumed to rely much more on textbook input than ESL users.

**6. Pedagogical applications**

In response to the results presented in this chapter, teachers may want to supplement textbook materials with their own corpus-informed materials. This could be achieved by using the web-based corpus platform Sketch Engine to create data-driven learning activities. Figure 1 shows how to search for MAKE causative constructions in the British National Corpus (BNC). Under the “Advanced” tab (a), Sketch Engine’s concordance function has a CQL “Query type” option (b). Having selected this option, the CQL queries formulated in 3.2 can be copied into the search field (c). If desired, the “Text types” drop-down menu (d) can be used to select a register subcorpus of interest, e.g., “Spoken demographic” (e) for everyday conversation.
<figure><img src="image_3.png" /><figcaption>**Figure 1:** Screenshot of the advanced concordance search function on sketchengine.eu</figcaption></figure>

Relevant concordance lines can then be selected and downloaded for students to explore causatives in real-life English (see Figure 2). Alternatively, tasks could involve students exploring a set of unsorted concordance lines and identifying which excerpts represent causative constructions. For detailed instructions and further ideas on how to use Sketch Engine to develop corpus-informed materials for the EFL classroom see Le Foll 2021b (for detailed instructions and further ideas on how to use Sketch Engine to develop corpus-informed materials for the EFL classroom, see Le Foll 2021b: chaps. 2, 5, 8, 9 and 15).
<figure><img src="image_4.png" /><figcaption>**Figure 2:** Screenshot of a random sample of query results with three selected concordance lines</figcaption></figure>

In addition to ENL corpora such as the BNC, Sketch Engine also hosts the Open Cambridge Learner Corpus (OCLC), which textbook authors, editors and EFL teachers can query to gain a better understanding of typical learner difficulties with constructions such as causatives. Figure 3 shows how, under the “Text types” (a) options of the OCLC, it is possible to narrow down the search to learners of a specific L1, e.g., Spanish (b), education level (c), CEFR level (d), etc.

<figure><img src="image_5.png" /><figcaption>**Figure 3:** Screenshot of the text type options when querying the OCLC</figcaption></figure>
Thus, different types of corpora can be used to design materials that supplement traditional textbook materials. In particular, corpus-informed materials can raise awareness of the most frequent semantic and lexical associations of each construction and any register-specific constraints. They can help pre-empt learners' frequent syntactic errors and unidiomatic lexical and semantic associations.

## 7. Conclusions

This study has examined the representations of causative constructions in school EFL textbooks, focusing on periphrastic constructions involving three of the most frequent causative verbs: CAUSE, GET and MAKE. The results have been compared to the findings of previous learner corpus studies with the aim of further exploring the input-dependent L2 acquisition hypothesis (Gilquin 2016) which suggests that EFL learners' use of specific constructions is likely to be influenced by the distributions of these same constructions in their main source of L2 input, namely their textbooks. Consequently, EFL teachers may want to draw on ENL corpus data to expose their students to constructions that are underrepresented in teaching materials (e.g., GET causatives). Additional, learner corpora can be used to raise awareness of frequent difficulties that learners of specific L1s face in attempting to produce particular constructions.

As in all corpus-based investigations, the observations made in this study are limited to the data included in the TEC-T and may only very tentatively be generalised to the entire population, in this case, of all lower secondary school EFL textbooks. Even more tentative are the hypotheses drawn from comparisons between this study's results and those obtained from learner corpora because they are based on different text registers. The learner corpora queried in Gilquin (2012; 2016), Altenberg & Granger (2001) and Liu & Shaw (2001) all consist of academic essays, a register which is not widely represented in secondary school EFL textbooks. Medium and register, however, have been shown to have a major influence on the distribution and phraseological patterns of causative constructions in English. For instance, the [X MAKE Y Vinf] construction has been shown to be about four times more frequent in spoken than in written language (Gilquin 2010: 226). Moreover, Gilquin (2012: 58) points out that [X MAKE Y FEEL] is highly frequent in spoken English, and demonstrates using distinctive collexeme analysis that the frequency of use of [X MAKE Y FEEL] in EFL academic essays is closer to its frequency in ENL speech than in ENL essay writing. It may therefore be tempting to conclude that school EFL textbooks are more representative of spoken than written language; however,
this hypothesis does not entirely hold, since it would also imply that causative constructions
with GET ought to be far more frequent than they were shown to be in 4.2. Register-based
variation ought to be considered when teaching causatives (see also Le Foll 2021a; 2022a;
2022b). Since current EFL textbooks do not do so adequately, Section 7 showed how textbook
authors and teachers can use Sketch Engine to explore how causative constructions vary in
different registers, e.g., academic writing vs. conversation.

References

Altenberg, Bengt. 2002. Causative constructions in English and Swedish: A corpus-based contrastive study. In Bengt Altenberg & Sylviane Granger (eds.), *Lexis in contrast: corpus-based approaches* (Studies in Corpus Linguistics 7), 97–116. Amsterdam: John Benjamins.

Altenberg, Bengt & Sylviane Granger. 2001. The grammatical and lexical patterning of MAKE in native and non-native student writing. *Applied linguistics* 22(2). 173–195.

Crossley, Scott A., Kristopher Kyle & Ute Römer. 2019. Examining Lexical and Cohesion Differences in Discipline-Specific Writing Using Multi-Dimensional Analysis. In Tony Berber Sardinha & Marcia Veirano Pinto (eds.), *Multi-Dimensional Analysis: Research Methods and Current Issues*, 189–216. Bloomsbury Academic.

Gabrielatos, Costas. 2013. *If-conditionals in ICLE and the BNC: A success story for teaching or learning? In* S. Granger, F. Meunier & G. Gilquin (eds.), *Twenty Years of Learner Corpus Research: Looking back, moving ahead*, 155–166. Louvain-la-Neuve: Presses Universitaires de Louvain.

Gilquin, Gaëtanelle. 2010. *Corpus, cognition and causative constructions* (Studies in Corpus Linguistics 39). Amsterdam: John Benjamins.

Gilquin, Gaëtanelle. 2012. Lexical infelicity in causative constructions. Comparing native and learner collostructions. In Jaakko Leino & Ruprecht von Waldenfels (eds.), *Analytical causatives: from ‘give’ and ‘come’ to ‘let’ and ‘make,’* 41–64. München: Lincom Europa.

Gilquin, Gaëtanelle. 2016. Input-dependent L2 acquisition: Causative constructions in English as a foreign and second language. In Sabine De Knop & Gaëtanelle Gilquin (eds.), *Applied Construction Grammar*, 198–259. Berlin, Boston: De Gruyter.

Goldberg, Adele E. 1995. *Constructions: A construction grammar approach to argument structure*. University of Chicago Press.

Goldberg, Adele E. 2005. Argument realization: the role of constructions, lexical semantics and discourse factors. In Jan-Ola Östman & Mirjam Fried (eds.), *Construction grammars: Cognitive grounding and theoretical extensions*, 17–43. Amsterdam; Philadelphia: John Benjamins.

Goldberg, Adele E. 2006. *Constructions at work: The nature of generalization in language*. Oxford University Press.
Granger, Sylviane, Estelle Dagneaux, Fanny Meunier & Magali Paquot. 2009. *International Corpus of Learner English*. Louvain-la-Neuve: UCL Presses Universitaires de Louvain.

Gries, Stefan Th. 2007. *Coll 3.2a package for R*. http://www.linguistics.ucsb.edu/faculty/stgries/teaching/groningen/index.html. (14 February, 2019).

Hausmann, Franz Josef. 1984. Wortschatzlernen ist Kollokationslernen. Zum Lehren und Lernen französischer Wortverbindungen. *Praxis des neusprachlichen Unterrichts* 31(4). 395–406.

Kilgarriff, Adam, Vít Baisa, Jan Bušta, Miloš Jakubíček, Vojtěch Kovář, Jan Michelfeit, Pavel Rychlý & Vít Suchomel. 2014. The Sketch Engine: Ten Years On. *Lexicography* 1(1). 7–36.

Langacker, Ronald W. 2008. *Cognitive Grammar: A Basic Introduction*. Oxford: Oxford University Press.

Le Foll, Elen. 2021a. Register Variation in School EFL Textbooks. *Register Studies* 3(2). 207–246. https://doi.org/10.1075/rs.20009.lef.

Le Foll, Elen. 2021b. *Creating Corpus-Informed Materials for the English as a Foreign Language Classroom: A step-by-step guide for (trainee) teachers using online resources* (Open Educational Resource). 3rd edn. https://elenlefoll.pressbooks.com. (30 July, 2021).

Le Foll, Elen. 2022a. *Making tea and mistakes: The functions of make in spoken English and textbook dialogues*. In Zihan Yin & Elaine Vine (eds.), *Multifunctionality in English: Corpora, Language and Academic Literacy Pedagogy* (Routledge Advances in Corpus Linguistics), 157–178. Oxon; New York: Routledge.

Le Foll, Elen. 2022b. *Textbook English: A Corpus-Based Analysis of the Language of EFL textbooks used in Secondary Schools in France, Germany and Spain*. Osnabrück University: Unpublished PhD thesis.

Liu, Eric T.K. & Philip M. Shaw. 2001. Investigating learner vocabulary: A possible approach to looking at EFL/ESL learners’ qualitative knowledge of the word. *International Review of Applied Linguistics in Language Teaching* 39(3). 171–194. https://doi.org/10.1515/iral.2001.001.

Mindt, Dieter. 1987. *Sprache, Grammatik, Unterrichtsgrammatik: futurischer Zeitbezug im Englischen* (Schule und Forschung). Frankfurt am Main: Diesterweg.

Möller, Verena. 2020. From pedagogical input to learner output: Conditionals in EFL and CLIL teaching materials and learner language. *Pedagogical Linguistics* 1(2). 95–124. https://doi.org/10.1075/pl.00001.mol.

Römer, Ute. 2005. *Progressives, Patterns, Pedagogy: A Corpus-Driven Approach to English Progressive Forms, Functions, Contexts, and Didactics* (Studies in Corpus Linguistics 18). Amsterdam: John Benjamins.

Stefanowitsch, Anatol & Stefan Th. Gries. 2003. Collostructions: Investigating the interaction of words and constructions. *International Journal of Corpus Linguistics* 8(2). 209–243.

Volkmann, Laurenz. 2010. *Fachdidaktik Englisch: Kultur und Sprache*. Tübingen: Narr.

Winter, Tatjana & Elen Le Foll. 2022. Testing the Pedagogical Norm: Comparing If-conditionals in EFL Textbooks, Learner Writing and English Outside the Classroom.
*International Journal of Learner Corpus Research* 8(1). 31–66.
https://doi.org/10.1075/ijlcr.20021.win.

Wong, S.C. 1983. Overproduction & Unidiomatic Usage in the Make Causatives of Chinese Speakers: A Cause for Flexibility in Interlanguage analysis. *Language Learning & Communication* 2(2). 151–165.
# Appendix I

To best compare pedagogical materials used in different educational systems, each textbook was labelled for proficiency level on a universal scale of A to E: level A textbooks correspond to the first year of EFL instruction at secondary level, in other words, beginner level to roughly A1 on the CEFR scale, whilst level E corresponds to the fifth year (CEFR B1–B2). Note that French textbook series only cover the first four years of secondary school (which take place at *Collèges*), which is why, whenever possible, a textbook from the same publisher corresponding to the fifth year of instruction (the first year of *Lycée*) was added. At the time of corpus compilation, Le Livre Scolaire did not produce any textbooks for *Lycées*.

The full bibliographic metadata is available on doi.org/10.5281/zenodo.4922819.

<table><caption>Table 5: Composition of the Textbook English Corpus (TEC)</caption><thead><tr><th>Country of use</th><th>Publisher</th><th>Textbook series</th><th>Volume</th><th>Level</th><th>Publication date</th></tr></thead><tbody><tr><td rowspan="12">France</td><td rowspan="4">Bordas</td><td rowspan="4">Hi There</td><td>6ème</td><td>A</td><td>2012</td></tr><tr><td>5ème</td><td>B</td><td>2013</td></tr><tr><td>4ème</td><td>C</td><td>2014</td></tr><tr><td>3ème</td><td>D</td><td>2015</td></tr><tr><td rowspan="2">New Mission</td><td>2nde</td><td>E</td><td>2014</td></tr><tr><td>6ème</td><td>A</td><td>2010</td></tr><tr><td rowspan="4">Nathan</td><td rowspan="4">Join the Team</td><td>5ème</td><td>B</td><td>2011</td></tr><tr><td>4ème</td><td>C</td><td>2012</td></tr><tr><td>3ème</td><td>D</td><td>2013</td></tr><tr><td>2nde</td><td>E</td><td>2010</td></tr><tr><td rowspan="4">Le Livre Scolaire</td><td rowspan="4">Piece of Cake</td><td>6ème</td><td>A</td><td rowspan="4">2017</td></tr><tr><td>5ème</td><td>B</td></tr><tr><td>4ème</td><td>C</td></tr><tr><td>3ème</td><td>D</td></tr><tr><td rowspan="12">Germany</td><td rowspan="4">Klett</td><td rowspan="4">Green Line</td><td>1</td><td>A</td><td rowspan="4">2006</td></tr><tr><td>2</td><td>B</td></tr><tr><td>3</td><td>C</td></tr><tr><td>4</td><td>D</td></tr><tr><td rowspan="4">Klett</td><td rowspan="4">Green Line New</td><td>5</td><td>E</td><td>2009</td></tr><tr><td>1</td><td>A</td><td>2014</td></tr><tr><td>2</td><td>B</td><td>2015</td></tr><tr><td>3</td><td>C</td><td>2016</td></tr></tbody></table>
<table><tbody><tr><td></td><td></td><td></td><td>4</td><td>D</td><td>2017</td></tr><tr><td></td><td></td><td></td><td>5</td><td>E</td><td>2018</td></tr><tr><td></td><td></td><td></td><td>1</td><td>A</td><td>2013</td></tr><tr><td></td><td></td><td></td><td>2</td><td>B</td><td>2014</td></tr><tr><td></td><td></td><td></td><td>3</td><td>C</td><td>2015</td></tr><tr><td></td><td></td><td></td><td>4</td><td>D</td><td>2016</td></tr><tr><td></td><td></td><td></td><td>5</td><td>E</td><td>2017</td></tr><tr><td rowspan="10">Spain</td><td rowspan="5">Richmond</td><td rowspan="5">Achievers</td><td>A1+</td><td>A</td><td rowspan="5">2015</td></tr><tr><td>A2</td><td>B</td></tr><tr><td>B1</td><td>C</td></tr><tr><td>B1+</td><td>D</td></tr><tr><td>B2</td><td>E</td></tr><tr><td rowspan="5">Cambridge University Press</td><td rowspan="5">English in Mind</td><td>Starter</td><td>A</td><td rowspan="5">2010</td></tr><tr><td>1</td><td>B</td></tr><tr><td>2</td><td>C</td></tr><tr><td>3</td><td>D</td></tr><tr><td>4</td><td>E</td></tr><tr><td rowspan="4">Oxford University Press</td><td rowspan="4">Solutions</td><td>Elementary</td><td>A</td><td>2014</td></tr><tr><td>Pre-Intermediate</td><td>B</td><td>2016</td></tr><tr><td>Intermediate</td><td>C</td><td>2017</td></tr><tr><td>Intermediate Plus</td><td>D</td><td>2017</td></tr></tbody></table>
Appendix II

<table><thead><tr><th>Register subcorpora</th><th>Example texts</th><th>Number of tokens</th><th>% of annotated tokens</th></tr></thead><tbody><tr><td>Isolated individual words, phrases or sentences</td><td>Exercises, vocabulary lists, gap-filling or phrase matching activities</td><td>914,083</td><td>34.09%</td></tr><tr><td>Instructional language</td><td>Instructions, grammar explanations</td><td>592,061</td><td>22.08%</td></tr><tr><td>Conversation</td><td>Dialogues, audio and video transcripts, speech bubbles</td><td>510,954</td><td>19.05%</td></tr><tr><td>Informative text</td><td>Fact boxes, newspaper articles, reports, webpages</td><td>302,916</td><td>11.30%</td></tr><tr><td>Narrative writing</td><td>Short stories, extracts of novels, story-like introductions to dialogues</td><td>253,888</td><td>9.47%</td></tr><tr><td>Personal communication</td><td>Blog and diary entries, personal letters and e-mails</td><td>67,098</td><td>2.50%</td></tr><tr><td>Poetry</td><td>Poems, songs, rhyme</td><td>26,194</td><td>0.98%</td></tr><tr><td>Other</td><td>Questionnaires, advertisements, formal letters, jokes, recipes</td><td>14,394</td><td>0.54%</td></tr><tr><td colspan="2">Total annotated tokens</td><td>2,681,588</td><td></td></tr></tbody></table>
Appendix III

word.freq: frequency of the word in the corpus

obs.freq: observed frequency of the word with/in causative_MAKE

exp.freq: expected frequency of the word with/in causative_MAKE

faith: percentage of how many instances of the word occur with/in causative_MAKE

relation: relation of the word to causative_MAKE

delta.p.constr.to.word: delta p: how much does the word/construction help guess the word?

delta.p.constr.to.word: delta p: how much does the construction help guess the
word/construction?

coll.strength: index of collocational/collostructional strength: -log10 (Fisher-Yates exact,
one-tailed), the higher, the stronger

<table><thead><tr><th>words</th><th>word. freq</th><th>obs. freq</th><th>exp. freq</th><th>relation</th><th>faith</th><th>delta.p. constr.to.word</th><th>delta.p. word.to.constr</th><th>coll.strength</th></tr></thead><tbody><tr><td>feel</td><td>1412</td><td>65</td><td>1.2691</td><td>attraction</td><td>0.046</td><td>0.2278</td><td>0.0453</td><td>88.7419</td></tr><tr><td>laugh</td><td>252</td><td>14</td><td>0.2265</td><td>attraction</td><td>0.0556</td><td>0.0492</td><td>0.0547</td><td>20.355</td></tr><tr><td>sound</td><td>344</td><td>10</td><td>0.3092</td><td>attraction</td><td>0.0291</td><td>0.0346</td><td>0.0282</td><td>11.8997</td></tr><tr><td>think</td><td>5432</td><td>25</td><td>4.8822</td><td>attraction</td><td>0.0046</td><td>0.0719</td><td>0.0038</td><td>10.3363</td></tr><tr><td>promise</td><td>87</td><td>4</td><td>0.0782</td><td>attraction</td><td>0.046</td><td>0.014</td><td>0.0451</td><td>5.8727</td></tr><tr><td>wonder</td><td>133</td><td>4</td><td>0.1195</td><td>attraction</td><td>0.0301</td><td>0.0139</td><td>0.0292</td><td>5.1389</td></tr><tr><td>seem</td><td>301</td><td>5</td><td>0.2705</td><td>attraction</td><td>0.0166</td><td>0.0169</td><td>0.0157</td><td>5.0425</td></tr><tr><td>sneeze</td><td>14</td><td>2</td><td>0.0126</td><td>attraction</td><td>0.1429</td><td>0.0071</td><td>0.142</td><td>4.1383</td></tr><tr><td>consider</td><td>122</td><td>3</td><td>0.1097</td><td>attraction</td><td>0.0246</td><td>0.0103</td><td>0.0237</td><td>3.7079</td></tr><tr><td>cry</td><td>139</td><td>3</td><td>0.1249</td><td>attraction</td><td>0.0216</td><td>0.0103</td><td>0.0207</td><td>3.5415</td></tr><tr><td>understand</td><td>654</td><td>5</td><td>0.5878</td><td>attraction</td><td>0.0076</td><td>0.0158</td><td>0.0068</td><td>3.4614</td></tr><tr><td>want</td><td>3280</td><td>10</td><td>2.948</td><td>attraction</td><td>0.003</td><td>0.0252</td><td>0.0022</td><td>3.0519</td></tr><tr><td>cringe</td><td>1</td><td>1</td><td>0.0009</td><td>attraction</td><td>1</td><td>0.0036</td><td>0.9991</td><td>3.0463</td></tr><tr><td>empathise</td><td>1</td><td>1</td><td>0.0009</td><td>attraction</td><td>1</td><td>0.0036</td><td>0.9991</td><td>3.0463</td></tr><tr><td>wince</td><td>1</td><td>1</td><td>0.0009</td><td>attraction</td><td>1</td><td>0.0036</td><td>0.9991</td><td>3.0463</td></tr><tr><td>stop</td><td>826</td><td>5</td><td>0.7424</td><td>attraction</td><td>0.0061</td><td>0.0152</td><td>0.0052</td><td>3.0072</td></tr><tr><td>happen</td><td>1387</td><td>6</td><td>1.2466</td><td>attraction</td><td>0.0043</td><td>0.0170</td><td>0.0034</td><td>2.759</td></tr></tbody></table>
<table><tr><td>itch</td><td>3</td><td>1</td><td>0.0027</td><td>attraction</td><td>0.3333</td><td>0.0036</td><td>0.3324</td><td>2.5696</td></tr><tr><td>sweat</td><td>3</td><td>1</td><td>0.0027</td><td>attraction</td><td>0.3333</td><td>0.0036</td><td>0.3324</td><td>2.5696</td></tr><tr><td>come</td><td>2638</td><td>8</td><td>2.371</td><td>attraction</td><td>0.003</td><td>0.0201</td><td>0.0022</td><td>2.5318</td></tr><tr><td>throw up</td><td>5</td><td>1</td><td>0.0045</td><td>attraction</td><td>0.2</td><td>0.0036</td><td>0.1991</td><td>2.3481</td></tr><tr><td>work</td><td>768</td><td>4</td><td>0.6903</td><td>attraction</td><td>0.0052</td><td>0.0118</td><td>0.0043</td><td>2.2696</td></tr><tr><td>ache</td><td>6</td><td>1</td><td>0.0054</td><td>attraction</td><td>0.1667</td><td>0.0036</td><td>0.1658</td><td>2.2692</td></tr><tr><td>suspect</td><td>6</td><td>1</td><td>0.0054</td><td>attraction</td><td>0.1667</td><td>0.0036</td><td>0.1658</td><td>2.2692</td></tr><tr><td>poke</td><td>7</td><td>1</td><td>0.0063</td><td>attraction</td><td>0.1429</td><td>0.0036</td><td>0.142</td><td>2.2024</td></tr><tr><td>stand out</td><td>7</td><td>1</td><td>0.0063</td><td>attraction</td><td>0.1429</td><td>0.0036</td><td>0.142</td><td>2.2024</td></tr><tr><td>disappear</td><td>136</td><td>2</td><td>0.1222</td><td>attraction</td><td>0.0147</td><td>0.0067</td><td>0.0138</td><td>2.1659</td></tr><tr><td>change</td><td>843</td><td>4</td><td>0.7577</td><td>attraction</td><td>0.0047</td><td>0.0116</td><td>0.0039</td><td>2.13</td></tr><tr><td>realize</td><td>168</td><td>2</td><td>0.151</td><td>attraction</td><td>0.0119</td><td>0.0066</td><td>0.011</td><td>1.9899</td></tr><tr><td>appear</td><td>172</td><td>2</td><td>0.1546</td><td>attraction</td><td>0.0116</td><td>0.0066</td><td>0.0107</td><td>1.9704</td></tr><tr><td>stretch</td><td>18</td><td>1</td><td>0.0162</td><td>attraction</td><td>0.0556</td><td>0.0035</td><td>0.0547</td><td>1.7944</td></tr><tr><td>appreciate</td><td>19</td><td>1</td><td>0.0171</td><td>attraction</td><td>0.0526</td><td>0.0035</td><td>0.0517</td><td>1.7711</td></tr><tr><td>flow</td><td>20</td><td>1</td><td>0.018</td><td>attraction</td><td>0.05</td><td>0.0035</td><td>0.0491</td><td>1.749</td></tr><tr><td>sign</td><td>22</td><td>1</td><td>0.0198</td><td>attraction</td><td>0.0455</td><td>0.0035</td><td>0.0446</td><td>1.708</td></tr><tr><td>react</td><td>242</td><td>2</td><td>0.2175</td><td>attraction</td><td>0.0083</td><td>0.0064</td><td>0.0074</td><td>1.6909</td></tr><tr><td>regret</td><td>24</td><td>1</td><td>0.0216</td><td>attraction</td><td>0.0417</td><td>0.0035</td><td>0.0408</td><td>1.6706</td></tr><tr><td>correspond</td><td>31</td><td>1</td><td>0.0279</td><td>attraction</td><td>0.0323</td><td>0.0035</td><td>0.0314</td><td>1.5608</td></tr><tr><td>grow</td><td>285</td><td>2</td><td>0.2562</td><td>attraction</td><td>0.007</td><td>0.0062</td><td>0.0061</td><td>1.5595</td></tr><tr><td>dream</td><td>47</td><td>1</td><td>0.0422</td><td>attraction</td><td>0.0213</td><td>0.0034</td><td>0.0204</td><td>1.3832</td></tr><tr><td>bite</td><td>54</td><td>1</td><td>0.0485</td><td>attraction</td><td>0.0185</td><td>0.0034</td><td>0.0176</td><td>1.3242</td></tr><tr><td>tidy</td><td>55</td><td>1</td><td>0.0494</td><td>attraction</td><td>0.0182</td><td>0.0034</td><td>0.0173</td><td>1.3164</td></tr><tr><td>stand up</td><td>63</td><td>1</td><td>0.0566</td><td>attraction</td><td>0.0159</td><td>0.0034</td><td>0.015</td><td>1.259</td></tr><tr><td>pay</td><td>420</td><td>2</td><td>0.3775</td><td>attraction</td><td>0.0048</td><td>0.0058</td><td>0.0039</td><td>1.2562</td></tr><tr><td>go</td><td>7419</td><td>11</td><td>6.6681</td><td>attraction</td><td>0.0015</td><td>0.0155</td><td>0.0006</td><td>1.1301</td></tr><tr><td>look</td><td>5761</td><td>9</td><td>5.1779</td><td>attraction</td><td>0.0016</td><td>0.0137</td><td>0.0007</td><td>1.106</td></tr><tr><td>rise</td><td>95</td><td>1</td><td>0.0854</td><td>attraction</td><td>0.0105</td><td>0.0033</td><td>0.0096</td><td>1.0868</td></tr></table>
<table><tr><td>believe</td><td>555</td><td>2</td><td>0.4988</td><td>attraction</td><td>0.0036</td><td>0.0054</td><td>0.0027</td><td>1.0475</td></tr><tr><td>cut</td><td>127</td><td>1</td><td>0.1141</td><td>attraction</td><td>0.0079</td><td>0.0032</td><td>0.007</td><td>0.9668</td></tr><tr><td>get up</td><td>146</td><td>1</td><td>0.1312</td><td>attraction</td><td>0.0068</td><td>0.0031</td><td>0.006</td><td>0.9099</td></tr><tr><td>hurt</td><td>160</td><td>1</td><td>0.1438</td><td>attraction</td><td>0.0062</td><td>0.0031</td><td>0.0054</td><td>0.8728</td></tr><tr><td>teach</td><td>169</td><td>1</td><td>0.1519</td><td>attraction</td><td>0.0059</td><td>0.003</td><td>0.005</td><td>0.8507</td></tr><tr><td>buy</td><td>773</td><td>2</td><td>0.6948</td><td>attraction</td><td>0.0026</td><td>0.0047</td><td>0.0017</td><td>0.813</td></tr><tr><td>improve</td><td>186</td><td>1</td><td>0.1672</td><td>attraction</td><td>0.0054</td><td>0.003</td><td>0.0045</td><td>0.8123</td></tr><tr><td>fly</td><td>198</td><td>1</td><td>0.178</td><td>attraction</td><td>0.0051</td><td>0.0029</td><td>0.0042</td><td>0.7874</td></tr><tr><td>prefer</td><td>219</td><td>1</td><td>0.1968</td><td>attraction</td><td>0.0046</td><td>0.0029</td><td>0.0037</td><td>0.7476</td></tr><tr><td>decide</td><td>1119</td><td>2</td><td>1.0057</td><td>attraction</td><td>0.0018</td><td>0.0036</td><td>0.0009</td><td>0.5745</td></tr><tr><td>bring</td><td>388</td><td>1</td><td>0.3487</td><td>attraction</td><td>0.0026</td><td>0.0023</td><td>0.0017</td><td>0.5306</td></tr><tr><td>forget</td><td>409</td><td>1</td><td>0.3676</td><td>attraction</td><td>0.0024</td><td>0.0023</td><td>0.0015</td><td>0.5116</td></tr><tr><td>fall</td><td>420</td><td>1</td><td>0.3775</td><td>attraction</td><td>0.0024</td><td>0.0022</td><td>0.0015</td><td>0.5021</td></tr><tr><td>hope</td><td>424</td><td>1</td><td>0.3811</td><td>attraction</td><td>0.0024</td><td>0.0022</td><td>0.0015</td><td>0.4987</td></tr><tr><td>love</td><td>1297</td><td>2</td><td>1.1657</td><td>attraction</td><td>0.0015</td><td>0.003</td><td>0.0006</td><td>0.488</td></tr><tr><td>keep</td><td>580</td><td>1</td><td>0.5213</td><td>attraction</td><td>0.0017</td><td>0.0017</td><td>0.0008</td><td>0.3907</td></tr><tr><td>wear</td><td>585</td><td>1</td><td>0.5258</td><td>attraction</td><td>0.0017</td><td>0.0017</td><td>0.0008</td><td>0.3879</td></tr><tr><td>sit</td><td>595</td><td>1</td><td>0.5348</td><td>attraction</td><td>0.0017</td><td>0.0017</td><td>0.0008</td><td>0.3823</td></tr><tr><td>stay</td><td>617</td><td>1</td><td>0.5546</td><td>attraction</td><td>0.0016</td><td>0.0016</td><td>0.0007</td><td>0.3705</td></tr><tr><td>spend</td><td>628</td><td>1</td><td>0.5644</td><td>attraction</td><td>0.0016</td><td>0.0016</td><td>0.0007</td><td>0.3647</td></tr><tr><td>guess</td><td>737</td><td>1</td><td>0.6624</td><td>attraction</td><td>0.0014</td><td>0.0012</td><td>0.0005</td><td>0.3143</td></tr><tr><td>run</td><td>758</td><td>1</td><td>0.6813</td><td>attraction</td><td>0.0013</td><td>0.0011</td><td>0.0004</td><td>0.3057</td></tr><tr><td>become</td><td>931</td><td>1</td><td>0.8368</td><td>attraction</td><td>0.0011</td><td>0.0006</td><td>0.0002</td><td>0.246</td></tr><tr><td>watch</td><td>974</td><td>1</td><td>0.8754</td><td>attraction</td><td>0.001</td><td>0.0004</td><td>0.0001</td><td>0.2336</td></tr><tr><td>help</td><td>2365</td><td>2</td><td>2.1256</td><td>repulsion</td><td>0.0008</td><td>-0.0004</td><td>-0.0001</td><td>-0.1921</td></tr><tr><td>listen</td><td>1587</td><td>1</td><td>1.4264</td><td>repulsion</td><td>0.0006</td><td>-0.0015</td><td>-0.0003</td><td>-0.2348</td></tr><tr><td>take</td><td>3177</td><td>2</td><td>2.8555</td><td>repulsion</td><td>0.0006</td><td>-0.0031</td><td>-0.0003</td><td>-0.3418</td></tr><tr><td>see</td><td>4185</td><td>2</td><td>3.7614</td><td>repulsion</td><td>0.0005</td><td>-0.0063</td><td>-0.0004</td><td>-0.5637</td></tr><tr><td>say</td><td>6719</td><td>3</td><td>6.039</td><td>repulsion</td><td>0.0004</td><td>-0.0109</td><td>-0.0005</td><td>-0.8394</td></tr></table>
<table><tr><td>use</td><td>5548</td><td>2</td><td>4.9865</td><td>repulsion</td><td>0.0004</td><td>-0.0107</td><td>-0.0005</td><td>-0.9086</td></tr><tr><td>make</td><td>4831</td><td>1</td><td>4.3421</td><td>repulsion</td><td>0.0002</td><td>-0.0119</td><td>-0.0007</td><td>-1.1678</td></tr><tr><td>read</td><td>5070</td><td>1</td><td>4.5569</td><td>repulsion</td><td>0.0002</td><td>-0.0127</td><td>-0.0007</td><td>-1.2452</td></tr><tr><td>get</td><td>5105</td><td>1</td><td>4.5883</td><td>repulsion</td><td>0.0002</td><td>-0.0128</td><td>-0.0007</td><td>-1.2566</td></tr><tr><td>do</td><td>19247</td><td>6</td><td>17.299</td><td>repulsion</td><td>0.0003</td><td>-0.0404</td><td>-0.0006</td><td>-2.8828</td></tr><tr><td>be</td><td>73499</td><td>1</td><td>66.0602</td><td>repulsion</td><td>0</td><td>-0.2326</td><td>-0.0012</td><td>-30.7971</td></tr></table>

In order to determine the degree of repulsion of verbs that are not attested with/in causative_MAKE, the following table gives the collocational/collostructional strength for all verb frequencies in orders of magnitude the corpus size allows for.

<table><thead><tr><th>absents. words</th><th>absents.ob s.freqs</th><th>absents.exp.fr eqs</th><th>relation</th><th>absents delta.p .constr.to.word</th><th>absents delta.p. word.to.constr</th><th>absents.collst lengths</th></tr></thead><tbody><tr><td>a</td><td>10</td><td>0.009</td><td>repulsion</td><td>0</td><td>-0.0009</td><td>0.5527</td></tr><tr><td>b</td><td>100</td><td>0.0899</td><td>repulsion</td><td>-0.0003</td><td>-0.0009</td><td>2.14</td></tr><tr><td>c</td><td>1000</td><td>0.8988</td><td>repulsion</td><td>-0.0032</td><td>-0.0009</td><td>0.0984</td></tr><tr><td>d</td><td>10000</td><td>8.9879</td><td>repulsion</td><td>-0.0321</td><td>-0.0009</td><td>2.9323</td></tr><tr><td>e</td><td>100000</td><td>89.879</td><td>repulsion</td><td>-0.3213</td><td>-0.0013</td><td>0.1344</td></tr></tbody></table>

If your collostruction strength is based on *p*-values, it can be interpreted as follows:

Coll.strength>3 => *p*<0.001; coll.strength>2 => *p*<0.01; coll.strength>1.30103 => *p*<0.05.