Running head: INDIVIDUAL DIFFERENCES

1

Why to treat subjects as fixed effects
James S. Adelman
University of Warwick
Zachary Estes
Bocconi University

Corresponding Author: James S. Adelman
Department of Psychology,
University of Warwick,
COVENTRY,
CV4 7AL,UK.
Telephone: +44 (0) 24 7615 0233

Electronic mail: J.S.Adelman@warwick.ac.uk

Z. Estes gratefully acknowledges the support of the Center for Research on
Marketing and Services (CERMES) at Bocconi University.

INDIVIDUAL DIFFERENCES

2
Abstract

J. S. Adelman, S. J. Marquis, M. G. Sabatos-DeVito and Z. Estes (2013) collected word
naming latencies from 4 participants who read 2820 words 50 times each. Their
recommendation and practice was that R2 targets set for models should take into
account subject idiosyncrasies as replicable patterns, equivalent to a subjects-as-fixedeffects assumption. In light of an interaction involving subjects, they broke down theinteraction into individual subject data. P. Courrieu and A. Rey’s (this issue)commentary argues that (1) single-subject data need not be more reliable than subjectaverage data, and (2) anyway, treating groups of subjects as random samples leads tovalid conclusions about general mechanisms of reading. Point (1) was not part ofAdelman et al.’s claim. In this reply, we examine the consequences of using the fixedeffect assumption. It (1) produces the correct target to check if by-items regression
models contain all necessary variables; (2) more accurately constrains cognitivemodels; (3) more accurately reveals general mechanisms; and (4) can offer more
powerful tests of effects. Even when individual differences are not the primary focus
of a study, the fixed-effect analysis is often preferable to the random-effects analysis.

INDIVIDUAL DIFFERENCES

3

We (Adelman, Marquis, Sabatos-DeVito, & Estes, 2013) evaluated the processes
involved in word naming using a new data set that was somewhat unusual for thefield: It had only four subjects, each reading 2820 words 50 times. This design’s
purpose was to permit us to treat systematic individual differences as replicablestatistical patterns requiring explanation. Since there was a significant subject × item
interaction, we broke down the interaction on a subject-by-subject basis. We
concluded (1) that current knowledge of the factors affecting reading falls well shortof accounting for all the systematic variance in the data; and (2) that when using
mega-study data, modelers have compared their models to variance-explainedtargets that are too lenient. Courrieu and Rey (this issue) argued that (1)single-subject data are not necessarily more reliable than subject-average data
(though on their empirical tests, they were in this case) and (2) anyway, averaging
over “randomly sampled” participants is more informative about “generalmechanisms of reading” (p. XX).
Our arguments did not rely on a comparison of the reliability of thesingle-subject data to the subject-average data — as assumed in their point (1) — but
rather on different approaches to estimating the reliability of subject-average data.
The difference between the two approaches results in analyses that correspond to the
traditional distinction between a fixed effect (ours) and a random effect (theirs) ofsubjects, or rather that of the subject × item interaction. We, of course, do not believethat subjects are fixed effects in the traditional sense (being exhaustive of the
population). However, we will demonstrate the counter-intuitive point that although
the fixed-effect analysis has narrower scope, it is more useful for assessing generalmechanisms, both for the comparison we performed and more generally. Whilst
different analyses can be useful for different research questions, the questions that are
scientifically or practically interesting in the context of experiments on word naming
are better answered by the fixed effect approach.

Decomposition of item varianceINDIVIDUAL DIFFERENCES

4

The fixed approach and the random approach differ in how they treat subject
idiosyncrasies, and different estimates result. Consider an experiment like ours withitems (I), subjects (S) and replicates of each subject-item (S × I) combination. The
variance to be explained is equivalent to the mean-squares associated with items,
MS I , which is an overestimate of the variance due to the actual item effect because
this estimate is contaminated by variance from subjects and noise. An item-R2 target
says what proportion of the MS I is due to the actual item effect. The random
approach assumes that the target of interest is the average of the population from which
the subjects are randomly sampled, whereas the fixed approach assumes that theobserved subjects are the target of interest. The random target is therefore
TR = 1 − MSS× I /MS I and the fixed target is therefore TF = 1 − MSW /MS I , where
MSW is the variance within each subject-item combination. TR is designed to treat
subject idiosyncrasies as to-be-left-unexplained and TF is designed to treat them as
to-be-explained. If one uses (as we did) a simple signal-noise dichotomy, i.e.
R2 = 1 − proportion noise, then things to-be-left-unexplained are noise. The random
approach is therefore equivalent to treating subject idiosyncrasies as (structured)
noise. E( MSS× I ) ≥ E( MSW ), with equality only when there are no stable individual
differences in item effects. If there are such stable differences, E( MSS× I ) > E( MSW ),and E( MSS× I /MS I ) > E( MSW /MS I ). That is, the noise estimate for TR is greaterthan that for TF . If — as we will show — MSW /MS I is the more useful definition ofnoise for the problems at hand, then MSS× I /MS I is an overestimate of the noise. This
is what we meant when we wrote “Analysis techniques that treat individual
differences as noise will necessarily overestimate the amount of noise contributing to
the mean RT for each word. This overestimation of noise results in anunderestimation of the variability that a model should explain, leading to an
overestimation of the success of models.” (Adelman et al., 2013, p. 1038). This
statement compares two analysis techniques (TR and TF ) that both generate targets
relating to MS I , that is, for the subject-average item means. Courrieu and Rey haveinstead generated and tested a hypothesis (Inequality 5) that compares theINDIVIDUAL DIFFERENCES

5

magnitude of TR for the subject-average item means with that of a single-subject
target (not TF ), infelicitously referring to this as a claim of ours to justify
single-subject experiments (the “second experiment” on p. 5). Their comparison is
not informative as to whether to use this single-subject target, because the total
variance of this measure is different to the measure with which it is compared (TR ).
We believe their confusion results because (a) this statement was part of ourdesign’s justification, but the part being justified was the use of replicates (required to
calculate MSW ); and (b) we did ultimately analyze the data on a per-subject basis.
Breaking down the subject × item interaction constrains models
Our analyses ultimately focused on analyzing each subject separately, because
we found a significant subject × item interaction, and an interaction limits theconclusions that can be drawn from the main effects. Therefore, we omitted to report
the overall TF of 93.22%, which is substantially greater than the TR that Courrieu andRey (this issue) report of 79.40%. It may not have been obvious that we were testing
the subject × item interaction because we used Kristof’s (1973) method instead of thestandard F = MSS× I /MSW . This F test analysis is of course also significant, but it
does not indicate if the interaction was driven by overall speed, so Kristof’s test wasmore informative (others might use a z-scoring approach to remove “general speed,”but see Adelman, Sabatos-DeVito, Marquis, & Estes, 2014, for criticism).
The formal argument that it is advisable to break down fixed-effect interactions
corresponds to an important substantive point: Individual differences add
information that is necessary to properly identify the constraints on general cognitive
mechanisms. Consider the length effect, which we considered separately for regular
words and for exception words. Such effects are sometimes believed to be absent fornaming short words on the basis of Weekes’s (1997) study, but mega-study datatypically show a small inhibitory effect. In our Figure 1, for exceptions, all showed
the inhibitory effect of length, except M, who showed no effect; for regulars, D and A
showed the inhibitory effect, M showed a facilitatory effect, and U showed no effect.
By taking the fixed approach, M’s idiosyncratic pattern adds information that is notINDIVIDUAL DIFFERENCES

6

available in the average data. It suggests that the length effect could be associatedwith a process whose influence is not central to reading. It also refutes any modelthat necessarily predicts an inhibitory length effect. In contrast, the random-effects
logic of “general psychology” as described by Courrieu and Rey would infer from asignificant average inhibitory length effect that a correct model would always predict
such an inhibitory length effect. Accounting for M’s pattern is not a simple matter ofadding idiosyncracies to such a model, but instead would require a different set ofmechanisms.
The fixed-effect model is the correct comparison for regression models
Most analyses of mega-studies use regression models on item means. Adelman
et al. (2013) calculated (a) such regression models and (b) R2 targets to compare themwith. The mega-study regressions are by-items, so they treat subjects as fixed effects.
A regression model of this sort will adapt its regression coefficients to match theidiosyncracies of its subject sample as well as possible, and its R2 will reflect how
well it has done so. The correct target for a regression model that takes subject
idiosyncracies into acccount (like those normally used on mega-studies) is a target
that takes subject idiosyncracies into account, namely our TF target. TR is lower andtherefore too lenient. We illustrate this (i) with a thought experiment for an ideal case,and (ii) with simulations of a more realistic set of data.
Illustration 1: Thought experiment
Consider some task for which there are two possible strategies. Strategy 1 isaffected (facilitated) only by variable A, and strategy 2 is affected (facilitated) only by
variable B. Stimuli are selected for a study in which A and B are not correlated, as in
an ideal experiment. Replicates are collected (each participant responds more thanonce to each stimulus), so that TF can be calculated. Each subject reliably uses a single
pure strategy, and there is no trial-to-trial variation in response times. That is, somesubjects’ RTs are perfectly correlated with A, and the other subjects’ RTs are perfectly
correlated with B. For a regression model with A and B as predictors of these averagedata, there are always some coefficient on A and B that can fit the average data

INDIVIDUAL DIFFERENCES

7

perfectly, so its R2 will always be 100%. The TF target is based on the idea that how
well each subject replicates himself is informative as to how well a model can do; there
is no trial-to-trial variation, each subject replicates himself perfectly, so TF is 100%.
This correctly reflects how well the regression model does. The TR target is based on
the idea that how well subjects replicate each other is informative as to how well amodel can do. When the sample contains some subjects using strategy 1 and someusing strategy 2, whose RTs correlate 0, the average correlation will be less than 1.
Then TR < 100% = R2 . TR is an underestimate that incorrectly indicates that the
regression is doing better than possible. Note that the average data offer no way todiscern the correct cognitive model. To address any concern that there may besomething peculiar to this ideal case that will not generalize, the next illustration ismore similar to the word naming situation we have been considering.
Illustration 2: Simulated data
We simulated artifical data from a model that contained the subjectidiosyncratic effects in the form of subject-specific coefficients in a linear model fromlexical predictors to response times. For simplicity, we included only three lexical
predictors, log. frequency (SUBTLEX + 1), length and a hypothetical to-be-discovered
variable with a standard normal distribution of scores over items, which we will callfoo. The subject idiosyncracies were simulated as follows: Frequency: a −3 × χ2 (5)distribution. Length: a N (6, 30) distribution. Foo: a 9 × χ2 (2) distribution.
For each simulation (representing an experiment), we sampled new subject
idiosyncracies and normally distributed trial-to-trial noise. Given the known noise
distribution, we used analytic results to produce item-mean R2 targets under fixedeffect and random-effect models, and compared these to the correct regression modelwith all three variables, and the “currently-known” regression model with only log.
frequency and length. TR was considered both with and without z-scoring of subjects.
We ran 1000 simulations, in which 4 subjects read each of 2712 items 50 times
each, using a noise standard deviation of 250. Across all simulated experiments, themean subject-average item R2 of the correct 3-variable model was 83.82%. The

INDIVIDUAL DIFFERENCES

8

average TF was 83.77%. The mean absolute deviation between the correct model’s R2
and TF was 0.35%. TF thus accuractly indicated what we should expect a model to do.
In contrast, the average TR was 66.84%, or 67.59% (with z-scoring), which are
underestimates of what the correct model achieves. The incomplete two-variable
model (without foo) explained 66.73% of the variance on average. TR would therefore
incorrectly label an incomplete set of predictors as adequate to explain the data.
The TR target leads to the wrong conclusion. It does so because — whether or
not one is presently interested in the subject variation — the subject variation in theeffects is systematically associated with cognitive processes that are of the same form
but different magnitude across subjects.
The fixed-effect model is the correct comparison for cognitive models
Just as multiple regression models have coefficients, cognitive models
explaining multiple lexical variables also have numerical parameters. These
parameters can be adapated to the idiosyncracies of a data set or indeed individual
participants to improve the fit. This is the same property of models that makes TF
rather than TR correct for regression models (see above). Although modelers
sometimes argue that their model parameters are not free, the qualitative mechanisms
are the theoretical content of the model, and the parameters could be altered to savethe theoretical content if the data required it. Indeed, given there are no truly random
samples, parameters should be optimized wherever possible (for further discussion,
see Adelman & Brown, 2008). So, TF is also the correct target for cognitive models.
The fixed-effect model is appropriate for some designs of experiment
The fixed-effect model for subjects can also be of use elsewhere. For cognitivemodeling, we are sometimes interested in whether a variable affects cognition, andthis is not logically identical to the effect being non-zero on average over the
population: a variable can facilitate for some people, inhibit for others and we would
want to account for it in a cognitive model, even if it averages out to zero. Clark(1973) considered a within-subjects between-items design. As well as rejecting F1 , herejected F2 , which uses a subject-fixed (item-random) model. He recommended less

INDIVIDUAL DIFFERENCES

9

powerful tests based on subject-random (item-random) models because he argued F2
produces significant results for a case (his (4b)) which it should not: When the effecthas zero mean but subjects vary around zero. But declaring such an effect notsignificant says — incorrectly — that the variable should not be accounted for by acognitive model. F2 might give a significant result in this case, and this would —correctly — indicate that a model should account for it.
General mechanisms are obscured — not clarified — by averaging over subjects
Statistical generalization of the average across experiments with new subjects is
not the same as scientific generalization. The statistical generalizations that Courrieu
and Rey (this issue) consider apply only to the mean of the population, but theregularities that govern cognition need not hold for the mean. If, for instance, practice
improves response latencies with a power law, but at different rates for each subject,
the average data will not show a power law, which is the regularity a cognitive model
should predict in this case (Estes, 1956). This problem holds for any cognitive
mechanism that is not mathematically equivalent to a linear regression model. As
such, individual differences should not be an afterthought in cognitive modeling.
The utility of mega-studies
Finally, Courrieu and Rey (this issue) suggest that our intent is to “disqualify”(p. 5) a wide range of mega-studies from publication. For the purposes of setting
model R2 targets and therefore directly testing cognitive models, we prescribe
fixed-effect analyses that require replicates. There is no disqualification ofmega-studies with replicates, if they are presented with the prescribed TF analysis,
nor without replicates, if they are used for any of the other purposes of mega-studies(see Balota, Yap, Hutchison, & Cortese, 2012, for numerous examples). That is:mega-studies without replicates should no longer be used for setting model targets,and mega-studies with replicates should use TF .
Conclusion
Courrieu and Rey (this issue) argued that single-subject data are not always
more reliable than subject-average data, but this was not our claim. They doINDIVIDUAL DIFFERENCES

10

accurately identify our article as stressing that investigating the structure ofindividual differences is critically important for assessing models. However, theyargue to the contrary that average data can model “general mechanisms.” The
random effects model this implies provides only limited information about thecognitive mechanisms that are generally true. Treating subjects as fixed effects results
in a more accurate picture of what a model can and should account for.
References
Adelman, J. S., & Brown, G. D. A. (2008). Methods of testing and diagnosing models:Single and dual route cascaded models of word naming. Journal of Memory andLanguage, 59, 524–544.
Adelman, J. S., Marquis, S. J., Sabatos-DeVito, M. G., & Estes, Z. (2013). The
unexplained nature of reading. Journal of Experimental Psychology: Learning,
Memory, and Cognition, 39, 1037–53.
Adelman, J. S., Sabatos-DeVito, M. G., Marquis, S. J., & Estes, Z. (2014). Individual
differences in word naming: A mega-study, item effects, and some models.
Cognitive Psychology, 68, 113–160.
Balota, D. A., Yap, M. J., Hutchison, K. A., & Cortese, M. J. (2012). Megastudies: What
do millions (or so) of trials tell us about lexical processing? In J. S. Adelman(Ed.), Visual word recognition, Vol. 1: Models and methods, orthography andphonology (pp. 90–115). Hove, England: Psychology Press.
Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language
statistics in psychological research. Journal of Verbal Learning and Verbal Behavior,
12, 335–359.
Estes, W. K. (1956). The problem of inference from curves based on group data.
Psychological Bulletin, 53, 134–140.
Kristof, W. (1973). Testing a linear relation between true scores of two measures.
Psychometrika, 38, 101–111.
Weekes, B. S. (1997). Differential effects of number of letters on word and nonword
naming latency. Quarterly Journal of Experimental Psychology, 50A, 439–456.