Second-language strategy instruction:
Where do we go from here?
Luke Plonsky
Northern Arizona University
Situating Strategy Use (SSU3)
October 15, 2019

Modeling L2 development
Learner EXTERNAL mechanisms/processes

Learner INTERNAL mechanisms/processes?

But which of these can we influence?

explicit
instruction
implicit

SA immersion
context
at-homeclass
Instruct. context
lab+/-amount
input
type

workingmemorySLenvironmentFLemotions

beliefs about
language learning
anxietyATI

techmedium
F2F
stress

L2 strategies
personality
relativeclauses

Languagecomplexity

aptitudeFSs

intelligencemotivation2

Individual differences(learner-internal variables associated with L2 development)

How strongly are these constructs associatedwith L2 proficiency / achievement (as shown
via meta-analyses)?
Probablynot
Probably, but very
little research
Probably, but very
little research

Yes!
Probablynot

Aptitude
WTC
Motivation
AnxietyStrategies
WM
L2MSS

0.49
0.48
0.37
0.36
0.31
0.26
0.22
0

Probably, but very
little research

Li (2016); Elahi Shirvan, Khajavy, MacIntyre, &Taherian (in press); Masgoret & Gardner(2003); Teimouri, Goetze, & Plonsky (2019);
Linck, Osthus, Koeth, & Bunting (2014); AlHoorie (2018).

0.1

0.2

0.3

0.4

Meta-analytic correlations (r)

0.5

0.6

Strategy Instruction (L2SI)
Def. Explicit training on specific practices or techniques that can be
employed autonomously to improve one’s L2 learning and/or use(Chen, 2007; Ellis & Sinclair, 1989; Tudor, 1996; Taylor et al., 2006).

(Discussed in over 400 empirical, theoretical, and review article and books (see here)

Outline
Part I: State of the science / substantive findings of SI- The WHATPart II: Methodological issues- The HOW

Part III: Recommendations

What do we know about theeffects of L2SI?
Part IStrategy Instruction• Intuitive appeal• Theoretical support• Strategic competence (e.g., Canale & Swain, 1980)• Learner-centeredness (Nunan, 1988; Tudor, 1996)• Developmental sequences (i.e., rate vs. route)• Autonomy / self-regulation / self-management (Gu, 2003; Rubin, 2005; Tseng,
Dörnyei, & Schmitt, 2006)“Teachers should not focus exclusively on the content of learning.
Instead, attention should also be given to the process. For, to be selfsufficient, learners must know how to learn.”From, Toward a Theory of Instruction (Bruner, 1966)

Critiques of Strategy Instruction & SI Research• Poor design (e.g., small sample sizes, non-random group assignment, exclusionof comparison groups)• Unjustified selection of strategies• Uncertainty of long-term effects• Lack of valid and reliable instruments• Incomplete reporting of treatments and results• Absence of a comprehensive theory (C’mon SLA folks!)• Cost/benefit ratio concerns“…what one must teach students of a language is not strategy, but language”(Bialystok, 1990, p. 147).

(Chamot, 2005; Dörnyei, 1995; Kellerman, 1991; Macaro & Erler, 2007; McDonough, 1995; Macaro &Cohen, 2007; Rees-Miller, 1993; Rose et al., 2018)

Reviewing Strategy Instruction(Chamot, 2005; Hassan et al., 2005; McDonough, 1995)• Contexts

Positive effects for SI…

• second language, foreign language• middle school, HS, university• Children, adults• beginner, intermediate, advanced• class, lab• Treatments• Strategies type: cognitive, metacognitive, socioaffective• Number of strategies: 1-99
• Short-, long-term: 1 day - 1 year
• L1, L2; teacher- or researcher-delivered• Outcomes• L2 skills: reading, writing, listening, speaking, vocabulary, grammar• others: autonomy, motivation, strategies use, general language ability

Reviewing Strategy Instruction(Chamot, 2005; Hassan et al., 2005; McDonough, 1995)• Contexts

Negative/mixed effects for SI…

• second language, foreign language• middle school, HS, university• Children, adults• beginner, intermediate, advanced• class, lab• Treatments• Strategies type: cognitive, metacognitive, socioaffective• Number of strategies: 1-99
• Short-, long-term: 1 day - 1 year
• L1, L2; teacher- or researcher-delivered• Outcomes• L2 skills: reading, writing, listening, speaking, vocabulary, grammar• others: autonomy, motivation, strategies use, general language ability

Plonsky (2019)
A meta-analysis of the effects of L2SI
RQs:
1. How effective is L2 strategy instruction?
2. What is the relationship between the effectivenessof SI and different learning contexts, treatments,and outcome variables (e.g., skill areas)?

First, what is meta-analysis?

Assumption:
Developing scientificknowledge is acumulative andcorporate enterprise.

• Empirical approach to reviewing literature• More systematic and objective than traditional reviews• Origin? (“Necessity is the mother of invention”)

First, what is meta-analysis?

Assumption:
Developing scientificknowledge is acumulative andcorporate enterprise.

• THREE hallmarks (Mizumoto, Plonsky, & Egbert, in press; Plonsky & Oswald, 2015)
1. Exhaustive (vs. selective) searches (sample ≈ population)• à Validity generalizability

2. Systematic coding for substantive features and effects(vs. subjectively or idiosyncratically interpreted)

3. Key component: effect sizes (e.g., d, r)

more precise, stable, intuitive, and informative (vs. p)• Consequently…

Meta-analyses provide stable, trustworthy
answers!
• Q: Does textual enhancement work?
• A: YES, but the effects are fairly small (d = .22); and it helps for grammar
learning but might impede text comprehension (Lee & Huang, 2008; K = 20)• Q: Is computer-based feedback helpful?
• A: Yes! Just as helpful or more so than face-to-face feedback (Ziegler, 2013;
K = 14).

Lyster & Saito (2010)• Q: Is it helpful to provide students with feedback when they make errors
in class?
• A: YES, but it depends on what type of feedback you provide• What about SI?

A meta-analysis of the effects of L2SI
(Plonsky, 2019)

RQs:
1. How effective is L2 strategy instruction?
2. What is the relationship between the effectivenessof SI and different learning contexts, treatments,and outcome variables (e.g., the four skills)?

Method – Inclusion criteria• Participants learning an L2• Treatment that included L2 strategy instruction• Data collected and compared in a control-experimental (between
groups) design• DV = quantitative measure of the effect of SI• Sufficient data reported to calculate an effect size (Cohen’s d)Method - Sample• 77 primary studies of the effectiveness of SI• 112 unique samples / treatment groups• 7,890 individual participants

Method – data collection and analysis• Coded for…
(a) substantive and(b) methodological features as well as(c) estimates of treatment effects (Cohen’s d).

• Analysis• RQ1: Weighted average overall• RQ2: Weighted average for subgroups created according tostudy features (i.e., potential moderators)

Results: RQ1• Overall effect size: d = 0.66 [.62, .69]
• What does this mean?
• Relative to SLA: “medium”

EffectSizedSmall-ish Medium-ish
25th
50th
percentile percentile
.40

.70

Large-ish75thpercentile
1.00

K = 346 primary studies and 91 metaanalyses of L2 research (N > 604,000)(Plonsky & Oswald, 2014)

Results: RQ1• Overall effect size: d = 0.66 [.62, .69]
• What does this mean?
••••

Relative to SLA: “medium”
Relative to L1 SI: d = .45 (Hattie et al., 1996)
Exp groups score on average 2/3 of an SD above control groups
Approximately 3/4 of EG participants outperform average CG participants(Lipsey et al., 2012)• Additional and practical considerations for interpretation••••

Teacher training
Materials development
Class time (cost/benefit ratio?)
Potential for long-term benefit?

Results: RQ1 à change over time?
• Overall effect size: d = 0.66 [.62, .69]
0.43

1980-2005 (k=70)

0.93

2006-2015 (k=42)
0

0.2

0.4

0.6

0.8

1

1.2

Effect size (d)Two possible explanations- “a notably greater standardization of intervention frameworks has gradually emerged in
the past decade” (Ardasheva et al., 2017)- Methodological (vs. theoretical?) maturity (Plonsky & Gass, 2011; Plonsky & Oswald, 2014)

Proficiency

Setting

Institution

Context

RQ 2: Effects of SI Across Learning Contexts
0.84

L2 (k=13)

0.57

FL (k=99)

0.77

Primary (k=11)

0.26

Secondary (k=28)

0.69

University (k=70)

0.55Class (k=78)

0.82

Lab (k=32)

0.39

Beginner (k=44)

0.74Inter/Adv (k=57)
0

0.2

0.4

0.6

Effect size (d)

0.8

1

Type

RQ 2: Effects of SI Across Treatment Types
0.56

Cognitive (k=92)

1# of strategies

LengthMetacognitive (k=35)

0.49≤2 weeks (k=50)

0.65>2 weeks (k=57)

0.86

1 (k=41)

0.58>1 (k=48)

0

0.2

0.4

0.6

Effect size (d)

0.8

1

1.2

RQ 2: Effects of SI Across L2 Skill Areas
1.11

Strategy use (k=10)
1

Speaking (k=13)
Reading (k=41)

0.82

Vocab (k=33)

0.63

Writing (k=8)

0.59

Listening (k=10)

0.06
2.07

Pronunciation (k=2)
0.75

Grammar (k=4)
0.05

General (k=5)
-0.5

0

0.5

1

1.5
Effect size (d)22.5Ardasheva, Wang, Adesope, & Valentine (2017)• Meta-analysis of the effects of L2SI on• RQ1: L2 performance• RQ2: Other self-regulated outcomes (e.g., anxiety, self-efficacy, attitudes)• 2008-2014 only
• Sample• RQ1: 39 reports (47 samples)• RQ2: 16 reports (17 samples)Ardasheva, Wang, Adesope, & Valentine (2017)
Overall Results
0.78

Language (k=43)

SR learning (k=17)

0.87

0

0.2

0.4

0.6

0.8

1

Ardasheva, Wang, Adesope, & Valentine (2017)
Results, RQ1(linguistic outcomes)
All (K=43)

0.78

Vocab (k=4)

1.23

Reading (k=20)

0.76

Listening (k=9)

0.68

General (k=2)

0.62

Speaking (k=2)

0.61
0.47

Writing (k=7)
Grammar (k=2)

0.13
0

0.2

0.4

0.6

0.8

Strategy use (k=10)
Speaking (k=13)
Reading (k=41)
Vocab (k=33)
Writing (k=8)
Listening (k=10)
Pronunciation (k=2)
Grammar (k=4)
General
(k=5)
1
1.2
-1

0

1.11
1
0.82
0.63
0.59
0.06

2.07

0.75
0.05
1.4
1

2

3

Ardasheva, Wang, Adesope, & Valentine (2017)
Results, RQ2(non-linguistic outcomes)
All (K=17)

0.87

Strategy effectiveness (k=20

1.26Strategy use (k=11)

0.98

Anxiety (k=1)

0.9

Attitudes (k=2)

0.54

Self-efficacy (k=2)

0.27
0

0.2

0.4

0.6

0.8

1

1.2

1.4

Additional meta-analytic evidence for strategies• English learning (overall) (Elahi Shirvan, 2014)• Reading (Chaury, 2015; Maeng, 2014; Taylor et al., 2006)• Vocabulary learning strategies for EFL learners (Nematollahi et al., 2017)• Web-based instruction (Chang & Lin, 2013)

Preliminary Implications and Discussion
SI can be effective in all contexts and for all skills but appears to bestronger:(a) with non-beginners (“threshold” in Chamot, 2016; “the rich get richer”?)(b) with metacognitive strategies(c) over longer periods of time, and(d) fewer target strategies (i.e., less is more)

BUT
A great deal of further research is still needed across…
- Learner demographics and contexts- Linguistic (i.e., skills) and non-linguistic domains (e.g., anxiety)- Individual strategies

The HOW (SI Methods)
Part IIWe have some issues• Design & Instrumentation (see e.g., Pawlak, 2019; Rose et al., 2018)• Small samples• Lack of delayed posttests• Lack of theoretical or empirical justification of strategies taught• Evidence of reliability (internal consistency) and validity often unknown
BOTH for measures of strategies AND L2 performance!

“You can’t fix with analysis what you
bungled by design”(Light et al., 1990)

No analysis—however sophisticated or
elegant—can make up for poor
instrumentation.

At least we’re not alone?
• True. These problems are pervasive throughout pretty much all of
applied linguistics (and throughout the social sciences)!

Reliability evidenceO(observation) = T(true score) + E(error)Reporting of reliability across domains of L2 research
Effects of L2 practice6
7

Nekrasova & Becker (2009)
Mackey & Goo (2007)
Norris & Ortega (2000)
Russell & Spada (2006)
Derrick (2016)
Brown (2016)
Jeon & Kaya (2006)
Plonsky (2011)
Ziegler (2013)
Plonsky (2013)
Adesope et al. (2010)
Lee, Jang, & Plonsky (2015)
Adesope et al. (2011)
Plonsky & Kim (2016)
Plonsky & Gass (2011)
Liu & Brown (2015)
WCF

16
20
28
37
38
41
43
45
46
47
50
59
64
66
0

20

40

60

80

100

What about the amount of (measurement) ERROR?
What’s typical for the field?
• Reliability generalization meta-analysis (RGM) (Plonsky & Derrick, 2016)• K = 537 from 16 L2 journals• 2,244 reliability estimates
0.82 [.74 - .89]

Instrument(K = 1,323)

25th Percentile
75th Percentile

0.92 [.83 - .96]

Interrater(K = 861)

0.95

Intrarater[.90 - .96]

(K = 40)

0.7

0.75

0.8

0.85

0.9

0.95

1

In other domains of L2 research?
• TBLT (K = 85; Plonsky & Kim, 2016)

In other domains of L2 research?
• L2 pronunciation (K = 77; Saito & Plonsky, 2019)

rel.=

0.93

0.87

0.86

0.76

N/A

N/A

What about for L2 strategies?
• Or even for different categories of strategies?

• We really don’t know!
• Why does this matter?

• Unreliability à Error à Threat to validity• Attenuation of effects (signal vs. noise)Validity EvidenceO(observation) = T(true score) + E(error)

What we say about validity• Chapelle (in press): “validation should be of central importance for the credibility of
research results”• TQ Author Guidelines: Authors should provide a “Description of the instruments, what
they are designed to measure, and a report of their validity to the extent possible, andtheir reliability.”• Ellis (in press): “While researchers have always recognized this issue [validity in SLA
measurement], they have largely ignored it, often happy to talk about learning with noconsideration of the type of data they had collected”• Norris & Ortega (2012): ”Problematic… is the tendency to assume—rather than build an
empirical case for—the validity for whatever assessment method is adopted (pp. 574-575).
• Schmitt (2019): “Most vocabulary tests are not validated to any great degree.”

What about strategy scales???
See seminal works by Cronbach & Meehl; Messick; Kane; Chapelle, Enright, & Jamieson

Are questionnaires to blame?
(Great examples of alternatives in Gu’s plenary andYashima & MacIntyre’s symposium)• 1. Indirect measures of the construct of interest• Suggestion: triangulation (e.g., +observations; +interview)• 2. Responses often limited to what is being asked• Suggestions: piloting; open-ended items; interviews• 3. Self-selection bias
• Suggestions: random or purposive sampling; missing data analysis• 4. Anonymity à +/- truthfulness?
• Suggestion: triangulation• 5. Response values? (3, 5, 9, 1,000?)• Suggestions: piloting; clear instructions; scale descriptors• 6. Quantification without consideration of numerical values• Suggestion: rich qualitative data; better use of stats• 7. Ambiguous (“double-barreled”) items• Suggestion: Pilot. Leave room for comments.

Potential
threats tovalidity

Validity = multifaceted

Construct

Discriminant/ Divergent

Face

Predictive

Convergent/ concurrent

To what extent does L2 research demonstrate an
explicit concern for (different facets of) validity?
“There is perhaps an unwritten agreement that readers will accept measuresused in an SLA study at face value without asking about their reliability andvalidity for the task at hand.” (Cohen & Macaro, 2013, p. 133; see Bachman & Cohen,
1998).

• Is this true in general?
• And for strategies research?
• Do you ever see validity evidence?

To what extent does L2 research demonstrate an
explicit concern for (different facets of) validity?
• How could we address this Q?
• Collect a representative sample of studies…
• Synthetic approach• Very time-consuming• Subject to high inference judgments• Corpus-based approach• Fast and objective• Valid?

To what extent does L2 research demonstrate an
explicit concern for (different facets of) validity?
• Second Language Research Corpus (L2RC; Plonsky, n.d.)• 22 journals• 22,363 articles (1946-2018)• 147,293,764 words• Searched for occurrences of:- [predictive, discriminant, divergent, construct, face, convergent, concurrent] + validity- validity argument

AL, ALL, AP, BLC, CMLR, ELTJ, FLA, IJAL, IRAL, JSLW, LAQ, LA, LL, LL&T, LTeaching, LTR,
LTesting, MLJ, SLR, SSLA, System, TQ

To what extent does L2 research demonstrate an
explicit concern for (different facets of) validity?
Construct

1.93

Face

0.71

Preditive

0.54

Concurrent

0.52

Discriminant

0.24

V Argument

0.19

Convergent

0.18

Divergent

0.10

Note scale0

2 in every 100 articles

How might the strategies
literature compare???

1 in every 1,000 articles

2

4

6

8

10

(What about false positives? False negatives?)

It’s not all bad!
• Nakatani (2006): Scale development/validation• Development of the Oral Communication Strategy Inventory (OCSI)• Stage 1: Open-ended questionnaire (N=80)• Stage 2: Piloted with 400 à (exploratory) factor analysis (item structure) à 8
categories for speaking and 7 for listening strategies• Stage 3: Compared with data from SILL (N=62)• Mizumoto & Takeuchi (2012): Scale validation• Self-regulating Capacity in Vocabulary Learning Scale (SRCvoc)• Study 1: N = 443 à item-analysis: ITC of >.4; alpha for subscales• EFA to examine factor structure• Study 2: N=914 à alpha for subscales; CFA• Ardasheva & Tretter (2013) : Validation of modified version of SILL
• Revision of items and piloting• Administered to 1057 child learners of ESL• CFA à 6 factor solution

It’s not all bad!
• See also
• Tragant et al. (2013)• Ardasheva (2016)• Teng & Zhang (2016)

Summary for Part II• What we need is a non-casual, rigorous, and systematic agenda
focused on measurement as it pertains to L2 strategies and strategy
instruction.

Looking ahead à L2SI research wish list
(see Sudina & Plonsky, in press)

What / substance• SI across all skill areas. Esp: writing, listening, pronunciation, test-taking• Aptitude-treatment interactions with L2SI

(e.g., with beliefs, working memory; see Yashima, Nishida, & Mizumoto, 2017)• SI for specific learning contexts: SA, CALL/MALL, EMI/CLIL• Teacher training• Studies of teacher beliefs regarding SI and• Effectiveness of teacher training interventions for SI• The role of strategic transfer (L1 à L2; L2 à Ln)

Looking ahead à L2SI research wish list
(see Sudina & Plonsky, in press)

How / Method: Designs• Non-”WEIRD” samples: e.g., SL, pre-adolescent, advanced learners• Validity evidence/arguments for the utility of individual strategies ßessential justification for L2SI studies but RARELY present• A clearer understanding of the long-term effects of SI• “Bigger” and more longitudinal designs—at the curricular level

Looking ahead à L2SI research wish list
(see Sudina & Plonsky, in press)

How / Method: Measurement• Validity arguments for measures of both(a) strategy usage (Takeuchi, 2019; Tseng et al., 2006) and- Situated, qualitative, and mixed methods (Pawlak & Oxford, 2018; Rose et al., 2018)- Macro+micro perspective (Pawlak, in press)- Scenario-based scales à +contextualization (see Teimouri, 2018)- Studies of the predictive validity of individual strategies and L2 performance(as a pre-requisite for SI)(b) L2 performance… all to them be made available on the IRIS database iris-database.org à+consistency across studies!!

Looking ahead à L2SI research wish list
(see Sudina & Plonsky, in press)

How / Method: Data report and analyses• More thorough reporting of• Sample characteristics (e.g., proficiency)• Treatments (e.g., length/intensity, materials)• Data (ESs, CIs, visuals, reliability coefficients)• More informed use of quantitative analyses(Mizumoto & Plonsky, 2015; Nix, 2018; Takeuchi, 2019)• E.g., Rasch analysis; Multivariate models; corrections for attenuation due tomeasurement error

Looking ahead à L2SI research wish list
(see Sudina & Plonsky, in press)

How / Method: Beyond individual studies• Replication studies!
• Additional meta-analyses of SI focused on individual strategies or skills(e.g., vocab, speaking)• Systematic review and meta-analysis of reliability coefficients (what’s
normal?)Thank you!
Luke Plonsky
lukeplonsky@gmail.com
lukeplonsky.wordpress.com