Archived Information

**Cognition and Student Learning Research Grant Program**

**Proceedings from the Pre-Application Meeting of**

**February 19, 2002**

**Washington, D.C.**

**Bringing the Science of Cognition to Educational Research and
Practice**

**Dr. Robert A. Bjork**

Dr. Reyna: I would like to introduce Dr. Robert Bjork, who is a
Professor of Psychology at the University of California at Los Angeles.
His research interests focus on how students learn and remember, and on
the implications of research for training and instruction. He's the past
president of the American Psychological Society, and he chaired a
National Research Council Committee on techniques for the enhancement of
Human Performance, 1988--1994, so he's been useful to the government in
the past. He's a fellow---and I won't go over all his honors or we'd be
here all day---he's a fellow of the Society for Experimental
Psychologists, of the American Psychological Association and fellow of
the American Psychological Society. He's served as president of the
Western Psychological Association, chair of the Psychonomics Society,
chair of the Council of Editors of the American Psychological
Association. And, he's a good teacher, too. I thought you'd be pleased
to know that. He received UCLA's Distinguished Teaching Award. Take it
away.

Dr. Bjork: Thank you very much. I'm really delighted to have the
opportunity to speak here today, because, indeed, I think this program
on cognition and student learning is important and timely. It's
important because, as people in this room know, the need is great.
Education is not as effective as it needs to be, should be, and can be,
in our nation, particularly math and science. And, education is the
future. Not only for the children of our nation but, clearly, for our
nation as a whole.

The proposal's timely, in my opinion, for a couple of reasons that have
to do with the particular juncture we're at. One is that, in my opinion,
the body of basic research on elementary cognitive processes, including
learning and memory processes, has reached the point where it provides a
foundation for educational interventions that have the potential to have
very dramatic effects.

Perhaps more important, it's timely because at this point in time, there
is a will to have this happen. Politicians of all stripes are behind
educational initiatives. And, it's not just politicians, it's parents,
the average citizen, employers, the business world---all have a feeling
by personal experience, example, that our education system is, in many
ways, not as effective as it should be, and that it's the highest
priority of national need.

The challenge, though, of bridging the gap between this basic research
and translated educational practice is also great. In my opinion, the
gap is huge right now between where research stands and what kinds of
principals often guide educational practice. In part, that gap is huge
because there has been so much progress in basic research across the
last couple of decades.

Another reason the challenge is great, is that bridging the gap demands
a level of cooperation and communication and good will between
researchers and educators, that, for a variety of reasons, has not
existed, and sometimes has been actively resisted. There's plenty of
blame on both parties for that lack of communication and cooperation.
But, part of the problem is the sort of sociology of science, where,
sort of working in this particular gap between basic research and
educational practice is often not rewarded to the same degree in the
university setting that the basic research itself is. Maybe not
welcomed. And, so, it is a challenge to bridge that gap.

Finally, as I'll really focus on today, I think it's a challenge,
because the ways that I see based on basic research to dramatically
upgrade instruction, are, in many ways, unintuitive, and counter to
prevailing practices. And, as a matter of fact, not necessarily likely
to be well received by teachers or students, at least initially.

We're at a period, really, that is, in my opinion, analogous to the
things that Russ Whitehurst in his comments mentioned with respect to
farming and with respect to medicine. It is the case, I think, that our
intuitions serve us poorly in terms of whether the conditions of optimal
learning, face validity serves us poorly, and standard practice often
does not incorporate the kinds of activities that best produce learning.

I was present when Russ Whitehurst gave a talk to the Council of
Scientific Society Presidents in Washington. A group that's mainly
physical, biological scientists, but behavioral scientists by certain...
A splendid talk where he made the same sort of arguments for a research
based approach to education. And, I mention that, in part, because,
among that set of people, which include Nobel Laureates and physics
people with extraordinarily impressive credentials, there's nothing that
interests that group more, or they think is more important, than early
math and science education. Nothing in their own field captivates them
as much, or do they think as important.

So, I think we've made massive kinds of progress in our understanding of
the cognitive processes that underlie how we learn and remember, or
sometimes don't learn and don't remember. I'm reminded that, over 20
years ago now, I was invited to be after-dinner speaker for the Harvard
Radcliffe Club of Southern California. Very nice event, because they
held it in a beachside resort in Santa Barbara. Our whole family was
invited to go up there. And, before the banquet, Elizabeth and I went
with our 2 young boys, the oldest then about 8. Went in to check out the
audio visual and so on equipment there in the banquet hall. And at some
point, we hear the loud speaker system go on, and there's our oldest son
who was about 8 at that point, looking out. Pretty good imitation of his
father, he said, "Ladies and gentleman, the memory is in the brain, the
brain is in the head. Any questions?" I later went on for about an hour
saying more or less the same thing.

I was also struck maybe a dozen years ago at UCLA. There was a program
called the Best of UCLA, where the really big donors go through a kind
of simulated day of 3 classes in the morning, lunch, 3 classes in the
afternoon. I was asked to do one of these. And, I was trying to think
and I sort of went to one of my graduate students and I said, my gosh, I
got 45 minutes to tell them about work on memory. What should I tell
them? The student said, "Tell them the more they help UCLA, the more
their memories will improve."

Well, I think we have more to say than that now. And, one thing I want
to focus on today is sort of the problem of knowing what we have
achieved as educators. With little or no exceptions, the goals of
instruction and training are long-term goals. As teachers, or trainers,
or learners ourselves, we'd like instruction to resolve in the kind of
changes in mental representation, the kind of linkages and so on, that
result in the kind of learning and comprehension that will transfer past
that particular instruction, to support additional learning, and that's,
in some ways, the whole idea of prerequisites. And, we'd like this
learning transferred to our real lives, to the job setting and so on.

The problem is that, as instructors, what we get to look at is
performance during training. And, that brings up something that goes
back to an old distinction in psychology, the distinction between
learning and performance. What we observe as performance, what we're
trying to infer, is learning. And, the former can be a notoriously poor
guide to the latter, as I'll illustrate in a few ways.

Now, there is an old tradition on learning versus performance in
psychology, and learning theorists of decades ago found that distinction
necessary for a number of reasons. One of which was the fact that, in
many situations, both human and animal research, you could see that
during a period when no apparent change in performance was happening, if
you then later introduced reinforcement or some other factors, you'd see
that a great deal of learning did happen, even though you couldn't see
it in the context of the performance. Some of you may know about latent
learning experiments and so on.

Now, recently, we have abundant evidence that the converse is true as
well. Little or no learning can happen in the presence of very
substantial changes in performance and apparent learning. Now, I want to
just summarize a few of those things, and then come back to some of the
more practical issues of trying to translate basic research to
educational practice.

So, on this list are things that I come to call desirable difficulties.
Because they're all things that have the property that they appear
during instruction or training to slow the learner's progress, sometimes
upset or possibly even discourage the learner, but then are things that
are associated with better long-term retention and transfer.

So, there are conditions of instruction that make the learner advance
rapidly, feel good, and show a striking rate of improvement on the
so-called learning curve, that do not support retention and then things
that appear to be causing difficulties. Now, I'll try to emphasize that
the word "desirable" is important here. I sometimes talk about these
things and later have people give it as a kind of justification, hah.
Well, it was a benefit to my being confusing in my lectures all my years
and so on. The "desirable" is important, because there's plenty of ways
to make things difficult that don't enhance learning.

Now, these things, varying the conditions of practice rather than
keeping them constant. Providing contextual interference. Well, I'll say
just a little bit about that. Distributing or spacing a practice on a
path, reducing feedback to the learner, rather than constant feedback,
using tests as learning events. This last one, there's so much to say
about this, and the importance of retrieval, production, generation,
testing, as facilitating the later ability to access knowledge, there's
so much to say about it that I've said all I'm going to say today.

Reducing feedback to the learner, I'll also skip saying anything about
today. Relatively recent research that seems to put the lie to the
notion that the more frequent, the more complete, the more immediate the
feedback to the learner, the better the learning. It appears that that
kind of excessive feedback leads to people not developing a sort of
internal representation of the knowledge or skill that needs to be
learned, and to use, instead, that feedback as a kind of crutch to be
better on the next trial. And, that's as much as I've said about that.

But, I will, I do want to give a few examples of these other kinds of
things. One thing I presume most or nearly everybody in this room knows
about are basic spacing effects. One of the most robust findings of
decades of experimental psychology is that you can enhance long-term
retention by spacing exposures to to-be learned material, which of
course, in our course outlines, we don't tend to do. We're organized, we
cover something, we're done with it. We go on to the next thing. We
don't have it come up later in the course, which is what these results
say we should do.

I just want to show this as an example of the basic problem that we face
in knowing whether we have achieved the kind of learning that we want to
achieve. This experiment is by Bahrick (Bahrick 1979). And, it involves
the subjects learning vocabulary. They had to learn the English
equivalent of 50 Spanish words. And, none of these subjects spoke
Spanish. And, the procedure was what is sometimes called an anticipation
method. They would show a Spanish word. The participants would try to
give the English equivalent. And, then they'd see what the English
equivalent was. And, they'd go through the whole list of 50 that way,
and then again and so on.

I won't go into much of the details of this. But, what was manipulated
in 2 experiments, one that involved 3 training sessions and another that
involved 6 training sessions, was the intersession or spacing interval
between the training sessions on these vocabulary items.

Now, I haven't shown session one, because performance at the start of
session one is zero. These people don't know. They just guess. This is
performance at the start of 2, start of 3, 4, 5, 6. Then the spacing
decisions were spaced either just with a momentary break. Everything
took place in one day. One day per training session, or 1 month, 30 days
between training sessions.

Now, what I want you to notice, and I made it hard for you not to notice
with these red arrows, is that when we look just at training, it
certainly appears that we cannot have this kind of 1 month interval
between sessions. Perhaps, because there's just too much forgetting of
what you learned before. You're best, in with the zero spacing, then
with 1 day, worst with the 30. But, Bahrick did an interesting thing. He
then let 30 days go by and tested all of the subjects. And, now you can
see at this criterion point, now you're best, if you had your sessions
spaced by a month apart, then 1 day, and then worst if they were all in
1 day.

Now, I put this up, because it's typical of a lot of the kind of results
that, if you imagine yourself being a teacher or trainer, in most
situations, what do you have access to? In most situations, you don't
see the performance in the next course. You don't see it on-the-job. You
don't see the post-experience performance. If this is what you got
exposed to, you would arrive at certain conclusions how to structure
training. When this longer term test, the more valid measure of
learning, provides a different kind of picture.

Another example now, providing contextual interference refers to a whole
set of manipulations, in which the material to be learned is
inner-weaved or manipulated such that, during the process, you maximize
the possible interference between the components of learning. Contending
with that appears then to enhance long-term memory and transfer. And,
I'll give you an example.

This is an example of something there's been a lot of work done on
what's called blocked and random practice. I want to use this Shea and
Morgan (Shea and Morgan 1979), because it's a nice little motor skill.
But, please, understand now that these blocked random practice results
that I'll mention briefly, have been shown with people learning logic,
operators have been shown to facilitate rehabilitation with stroke
victims, with a great range of things. A quite general principle.

In this experiment, the subjects had to learn 3 different movement
patterns. Each one of them, which could be signalled up on this
apparatus, involved the subject letting go of the start button, grabbing
this tennis ball, knocking over 3 of these barriers in a prescribed
order, and then putting the ball back in the second hole, all as fast as
they could. They might have to grab the ball, knock the middle one
there, that one there, that one there, and put it back. And, then they
had 3 different patterns that differed in which barriers to knock down.

Now, one condition, called blocked practice, they did what really, based
on my experience in the context of the NRC Committee and many other
places, is basically what most instructors, trainers would do without
thinking. Basically, you have them learn one of these. When they reach
some mastery, you learn the other. When they reach some mastery, you
learn the third. So, they got 18 trials on one pattern. 18 on another,
18 on a third. The other case, the random practice case, they got the
same 54 trials, 18 on each, but each trial was determined randomly which
1 they should practice.

Now, what I'm showing here is movement time. And, you can see now that
during these blocks of 6 trials on a given pattern, it appears that it's
sensible to block and avoid the interference. You're better, faster. In
the blocked case, the difference gets narrower, but, to the end of
training, you remain faster.

But, look what the picture looks like at 10 days. Now at this time, what
the second letter denotes is whether these final testing conditions were
done under random or blocked practice. And, you can see now that, if you
had blocked practice, and then test it at 10 days under random, it's
almost like you didn't have any training, in terms of your performance.
If you were trained under random, tested under random, performance holds
up well. Just as important, if you test under blocked practice, what
little difference there is, still favors random practice.

I show this again, because were this all that we would have available to
us, as teachers and..., we would come to a conclusion that's not
justified in terms of what long-term performance looks like. Now, I
mention this is quite prevalent, and not just in motor skills. It's been
shown with extra batting practice, incidentally, with varsity baseball
players.

But, it pops up in places that, in some ways, we wouldn't even think
about. So, for example, do you recognize this? I find in showing this,
this is a practice sheet for writing capital letters. It shows how to
make the strokes for W. And, a student makes one and then makes another
one. So, this is for writing practice penmanship. People above a certain
age have a great sense of familiarity to this. Below a certain age, at
some point, the educational system gave up on that, at least
temporarily. I sometimes feel that those of us who had to do it were
doubly punished. We had to do this, and we later had to read the papers
of people who didn't have to do it.

But, in any case, notice something about this, which is what Diane
Ste-Marie and her students noticed (Lemire and Ste-Marie 1996). The
designers of this workbook just thought it was only natural to have
these students do what kind of practice? You make a W, another one,
another one. Nine in total. Then you move to some X's and so on. So,
they changed this to introduce conditions of random practice. And, in
fact, found far better transfer. And, particularly, for writing that was
a matter of alternating letters.

We don't actually have a great deal of need in our writing to make 9 W's
in a row. Three maybe. World Wide Web. Sorry, I couldn't resist.

Now, this again, it's just a matter of showing a sort of generality.
Now, how about variation? Well, there's many things I could show you
that indicate that varying the conditions of instruction, bringing up
things, critical concepts in more than one context, and so on, enhances
particularly transfers.

And, one just as something to show, which I can show easily, that's
quite remarkable, this experiment with Kerr and Booth (Kerr and Booth
1978), which was actually done in a recreational facility, with 8
year-olds and 12 year olds. The results are the same, so I'll focus on
the 8 year olds. The task they had, in this experiment, is that children
were in a kneeling position. They had a box of miniature bean bags right
by them. And, their task was to throw the bean bag as close as they
could to the middle of the target on the floor, which is a 4 by 4 black
square. The thing that was made tricky about it was right before they
threw, a screen was pulled down to occlude their vision. They threw, and
then they could see how well they did.

Now, I'm not going to show you performance during the training. Oh, I
have to say what the manipulation is, don't I? My proposal would have
gotten rejected right there. Okay. One group practiced at a fixed
distance for the 8 year olds it was 3 feet. The other group had just the
same number of throws, but never at 3 feet, at a sort of random mixture
of 2 feet and 4 feet. But, the same number of throws. Then, during that
training, as you wouldn't be surprised, the people, the kids throwing at
the constant distance were the best. Quite a bit better than the kids
who had to vary. But, then after delay, they tested both groups at the 3
foot distance. Now, your full intuition says that's what the one group
did their whole time on. They're going to be better. But, no. There is
something about varying between this mixture of 2 and 4 feet, perhaps
learning a sort of parameter of the motor skills, that overrides that.
And, this is distance from the center of the target. So, these kids who
got 2 and 4 feet, but never 3 feet, were better at the 3 foot distance
than was the group that did all their practice there.

Now, I've been very tempted, living in Los Angeles, to write the Los
Angeles Lakers and say, have you tried having Shaq try a mixture of 14
and 16 feet \-- \[laughter\] \-- doing thousands of free throws in
practice, maybe it would help and maybe it wouldn't.

Now, as far as variation, those are simple motor tasks, but they're
conceptual equivalents. An important study in my experience, in my
opinion, indicative of important factors of learning, is one by Mannes
and Kintsch (Mannes and Kintsch 1987). They had all the participants
have a certain period of time to learn of an interesting, but technical
article on industrial uses of microbes, with the interesting title,
"Industry in Torment."

Now, half the kids, actually, these were college students, got a
consistent outline. Namely, they studied in advance an outline that
corresponded in the Roman Numeral I, A, B, kind of structure to the
article itself. That, in educational research, is often called an
advanced organizer. And, as probably some of you in this room know,
that's something that has such high face validity, that it's used and
it's persisted, even though, if you look at evidence, it's been hard to
get convincing evidence for.

This other group, this outline incidentally had 25 percent of the
factual information contained in the article itself, so did this
inconsistent outline. Now, inconsistent here doesn't mean inconsistent
at any level of the factual information. It means it had a different
structure. A structure that came from the Encyclopedia Britannica
article.

So, they got the same time to do this. And, then, at 2 different
intervals, and I'm not going to show you the actual data, they tested
their memory in different ways. A very consistent pattern. If it was
anything like verbatim recall, the people with the consistent outline
did better. It was something like the ability to draw, answer inference
questions, the ability to rate possible solutions to industrial
problems, then the people with the inconsistent outline did better.

Now, towards the end of my comments, I'm going to talk on the crucial
kind of meta-cognitive part of this. But, as a kind of preview, suppose
right after this article, we had asked, they didn't do this, but suppose
we'd asked both groups, "How helpful was that outline you got in terms
of your learning?" What would they say? These people would say, "Very
helpful." These people would say, "Are you kidding? That thing
mismatched," and they would give it a negative score.

Now, I'm going to skip over a second of another study out of the Kintsch
laboratories (McNamara, Kintsch, Songer and Kintsch 1996), but it makes
the important point relative to introducing manipulations that engage
the type of processes of trained ... learning. This is an experiment in
which the students studied materials about types of heart disease, 8th
grade students. And, they divided the students into 2 groups, matched in
every way as far as ability and so on. Except that one half of these 8th
graders had had a relevant biology course. And, the other group hadn't.
And, what they showed them in this text, not that you'll really be able
to read this, they had them study either a fully coherent version of the
text on heart disease, which has nice headings, linkages, the underlying
things, or transition kind of materials, and so on. Or, they had them
study a so-called minimally coherent version, in which those materials
were left out, and, basically had to be generated by the learner.

Now, if you, again--- I said I wasn't going to show this, and here I'm
showing it---when you look at basic text recall, the high coherent text
was better than the low coherent for both the high knowledge, this isn't
ability, now, just whether they had the background or not. And, the low
knowledge participants. But, what was striking was that if you gave a
thing that wasn't verbatim recall, but had to do with problem solving
questions, bridging inference questions, now, the low knowledge people
are still better with the high coherent text. But, notice that the high
knowledge subjects are better with the low coherent text. They're better
in the case where they did the active kind of process to provide or
generate the linkage organization.

But, notice what this says to us, as teachers. It says it's fine to talk
about desirable difficulties and engaging the processes, then, that
create learning. But, our students have to be in a position to do it.
The low knowledge without that biology background were not in a position
to do these kind of activities of generation production linking that
would be so effective for learning.

So, there's a matter here of tuning. We want to engage certain
processes, but our students have to be in a position to do that.

Let me now return to the meta-cognitive point I alluded to briefly.
Baddeley and Longman (1978) did an interesting study years ago, in
cooperation with the British postal system. The British postal system
was changing over to a postal code, and there was a need in their eyes
to train a large number of postal carriers, workers, in keyboard skills.
And, Alan Baddeley and a student of his, cooperated in terms of carrying
out research on how that was best done. One of the manipulations they
did was a very basic spacing manipulation. The postal carriers either
got one hour of practice once a day, one hour of practice twice a day, 2
hours of practice once a day, 2 hours of practice twice a day. Now, in
terms of hours to learn to master the keyboard, and in terms of a
measure of long-term retention that I'm now showing, it was a beautiful
replication of the spacing effects found in the experimental literature.
The more spaced was your practice, the more efficient it was per unit
time. But, they did something else, and may have been the first to do
it, though we and others have been doing quite a bit of research like
this. They rated, they had the people in the groups rate their mean
satisfaction with how effective they thought the training was. On a one
to 5 scale, from very satisfactory to 5, very unsatisfactory. What do we
see here? Almost a perfect negative correlation with actual learning.

So, the important point here is it's not just, the problem is not just
that we, as teachers or trainers, could be fooled by what we see in the
way of performance during training and instruction. The learner himself,
or herself, is also fooled.

Recently, for example, a student of mine, Dominic Simon, and I, (Simon
and Bjork 2001) did something very much like the Shea and Morgan
experiment I talked about where you had to knock, pick up the tennis
ball and knock the barriers down. The only thing was that in this era of
easy access to computers, we translated the task to hitting a sequence
of keys on a number pad. The tricky thing was you had to learn a given
sequence that indicates the key you started on, on the number pad. You
had to hit those keys in that order, not so tough. But, you had to do it
in 900 milliseconds, much harder. This pattern, in 1200 milliseconds.
This one, in 1500 milliseconds. Otherwise, then, they got blocked or
random practice, just as I talked about before. And, our basic results
just replicate the results I showed you before, quite neatly. Blocked
practice is better during the training process. On the retention test 24
hours later, which people were informed of, they came back, they first
tried to draw the pattern, and then they were tested, the random
practice was better.

But, what we did that was different is that after every 5 trials in a
given pattern, we interrupted things and we told our subjects, "Now, if
training stopped here, how well do you think you will do on this pattern
when you're tested tomorrow?" So, they were told about the test that
would come. And, during the training process, on every trial, they saw
what pattern, they responded, then they got feedback as to how close
they were in time. And, on a prediction trial, they would say, "Well,
how close do you think you will come on this pattern tomorrow?"

Now, were these participants able to discount in that prediction the
temporary advantages of blocked practice? You probably know, just
because I'm showing you, is that they weren't. Their predictions went
with their current practice, and mismatched totally their actual
performance, 24 hours later.

Now, this creates a significant problem for us. And, from the standpoint
of motivation and evaluation. I will tell you right now, that if I were
to draw on everything I know, and to take a given course, I even teach
the college level, draw on everything I know to try to create the kind
of experience to maximize long-term learning and transfer, I won't get
UCLA's teaching award, not based on my course evaluations. Because, in
terms of these things that can enable instruction, our impressions get
dominated by the sense of ease or fluency or our own subjective rate of
feeling like we're making progress. And, so, there's a challenge here,
not just from the standpoint of optimizing, drawing on research to
optimize the condition of instruction in some unintuitive, non-standard
ways. There's a kind of socialization of the learner himself or herself
that's necessary. When, in fact, I do do a variety of these things in
one course I teach at UCLA. It's a course on learning and memory, and
I've got time to run demonstrations and say why, they end up convinced,
and, presumably, I even change the way that they manage their own
learning after that.

But, that poses a basic challenge. And, at the root of the problem,
essentially, is that we have a kind of faulty mental model of ourselves
as learners. For whatever combination of reasons, the trials and errors
of everyday living and learning don't seem to inform us as to how the
system works. It's one of the reasons that research is so essential.

One possible reason to misunderstand ourselves, this is just conjecture
on my part, is that, as poorly as we may understand devices like tape
recorders or the memory in a computer, we understand them a lot better
than we do the far more complicated functional architecture of the human
brain. We're far more sophisticated, far more complicated. And, to some
degree, then, we think we work like some sort of recording device. I sit
out there as a student. The person who's got the knowledge talks. I
write it down. It writes itself on me.

I was struck, incidentally, you know, just flying here recently, and
going into that kind of comatose state you sort of go in right when
they're trying to tell you how to get out of the plane and so on. So, I
was just thinking sort of idly how many times I'd seen this
demonstration of inflatable life vests. Fifty, one hundred, whatever.
Well, I closed my eyes and said, "Do I know how to do it?" So, I closed
my eyes and I just kind of got one of these sort of salient reveries. I
imagined the plane was down and there was some smoke and those lights
on, and people started screaming. Where's the thing? Where is it? Up
there? That's the mask. I'm drowning. I don't need the mask. I need the
life---under the seat, right. But, like, where, is it in a packet, do
you zip it out, I mean. So, I got this kind of panicky, but then I
assumed that I found this thing. And, then you put it over yourself,
right, you hold it up. But, like, what's front and what's back in that
thing? Well, I assumed that I put it over myself and then you pull on
things, right? Hopefully, if you put on the right side, they're here and
not back there. So, you pull on, now what happens if that doesn't work?
So, I imagined it didn't work. You find somewhere, you find the tube.
And, again, better not be back there. But, anyway, you pull it. So, then
I imagined, I blew this thing up, went down to get out the window, and
what? You don't inflate it until you get out of the plane. So, it
doesn't look to me like you would have been much better at this than I
was.

Now, my contention, the relevant part of the story is, I would say, in a
talk actually at the Air Traffic Control College in Oklahoma City, I
once made this suggestion. If there was a place in the airport where,
when you had time to kill, you could go in there one time, take it out
and put it on, I think that's worth more than the 50 to a 100 passive
observations. We don't learn as recording devices. We learn as active
participants making connections, making linkages, and so on.

So, one more thing to show you on the recent result on sort of
understanding ourselves or the failure thereof. This is something by
Benjamin, Bjork and Schwartz, we published recently (Benjamin, Bjork and
Schwartz 1998). The participants in this experiment go through a series
of easy general knowledge questions. Who was the first president of the
United States. Not all that easy, but pretty easy. What we're interested
in is how fast they can answer, the latency to present the answer
button. And, they go through 20 of these. But, before this starts, we
tell them that after these 20 questions and after a brief period in
which they'll be given a distracting task, we're going to give them a
blank sheet of paper, ... held it like this. And, they're not going to
get the questions again. We're going to ask you to try to write down as
many of the 20 answers you provided as you can remember. And, so, during
this task, as soon as they entered one, we had them predict the
likelihood they would recall that particular answer later.

Now, what we knew from some past research, well, I know things I know
well now. I will answer well later. And, that's not, overall, a wrong
heuristic, provided that each type of test is the same. But, we don't
have in the average person, the kind of mental ... as learners members
that would make the sort of distinctions as between semantic memory and
episodic memory.

So, there's a whole set of these conditions that have the potential to
mislead the learner. Mass interblocking practice on a given task.
Keeping the conditions of practice constant. Providing continuous
feedback. Making the learner observer imitator, as in the life vest kind
of example. All of these things have the potential to make the learner,
feel, think, that progress has been made, learning has happened, when
perhaps, it hasn't.

And, in fact, as learners, we can be very subject to illusions and they
often take the form of over-confidence. So, for example, for a while, I
was consulting at a nuclear training facility in accident analyses and
in that context and other ones, the chief single factor in major
accidents, near accidents, is that somebody didn't possess knowledge,
skills, they thought they did or their supervisor thought they did. We
can be very prone to over-confidence when the conditions of practice
amount to crutches, predictability, massing, and stuff that prop up
performance. And, then those crutches are gone when we're on the job at
a later time and performance falls apart.

To just kind of wrap-up that argument about illusions, I think we're all
aware that, in the perceptual domain, that we have some stunning
illusions. And, this one is about, due to Roger Shephard, I think is
about the best I've ever seen (Shephard 1990). So, you're looking at
this and say, "What's the illusion, he's not going to try to tell me
that those 2 table tops are the same, is he?" Well.

I think you'll agree that I'm pretty much on top of that one. Now,
watch. Where's the ... \[Laughter.\] You know, one reason I think
there's, if you have the subjective percept that I did, one reason I
think there wasn't as many oohs and aahs, is we're all so computer savvy
these days, thinking that things morph into it. So, you think they're
different after all. It's just that when I turn this thing, look, it
gets taller and skinnier. It just morphs to fit that thing.

Anyway, I think, in the domain of how we learn, how we remember, that,
in fact, there are illusions that are that profound.

And, so, we try to think of impediments to the goals of this initiative.
To the critical goals of this initiative. To link up and to bridge
between basic research and actual educational practice. There are
obstacles of almost every kind. But, the need is great, the will is
great, and I think with the kind of cooperation, with the kind of
efforts, I think we can bridge this gap. There's a kind of notion of
readiness. I think educators are ready, in the trenches are more ready
to cooperate than they've been. People like myself and others who've
been buried in our laboratories with our simple materials, are eager to
try to make a difference. And, it's an important thing, not just in our
classroom context.

I have recently been contacted, I need to pronounce her name correctly,
Helen Abdobsi\[?\], she's a senior evaluation officer with the World
Bank. And, she's read some things that I'd written recently. And, she's
wondering if there's some relevance to what they're seeing. The World
Bank has funded literacy programs all over the world during the
nineties. These programs are accompanied by very positive effects of the
subjective kind, a sense of empowerment by people in them, a sense of
competence, very positive. The only problem is they can't see a lot of
literacy coming. So, the final test in terms of actual literacy
achievement has been almost uniformly disappointing. She's wondering how
it is then, the participants in this go through this literacy thing and
gain a sense of empowerment, a sense of competence that is in fact, not
there. It's likely that some of the same factors enter in.

As kind of a final point, because I really don't want to cut short the
time for questions today, when you try to think of impediments to
upgrading instruction, there are also, in addition to the types I talked
about, there are prevailing societal attitudes that get in the way as
well. In my opinion, there is a dramatic overtendency, and
overappreciation in our society, to attribute differences in performance
to differences in innate ability between individuals.

Companies spend far more on selection devices to find people with the
right stuff than on training. It's almost, if we had a thorough
discussion of how much is experience and how much is heredity and so on,
that gets very complicated. But, what I feel confident in saying is that
whatever the truth of the mixture, in our society right now, there is a
terrific overappreciation, overattribution, to innate differences. And,
a wild, underappreciation of the power of practice, experience, and the
basic capability of all of us to learn.

We come equipped with this remarkable kind of ability. And, instead,
what can happen is one early test in some math course, by some one
student, they label themselves then as that's not their thing and they
gear the rest of their life to make that be a self-fulfilling prophecy.
Ethnic groups, families, decide their thing is not good at this or good
at that. And, what it does is, it nullifies this remarkable ability that
almost all of us to learn in all sorts of domains. And, if you could
change the culture to just have people understand the potential to
learn, given the chance and given, bringing to bear the right kind of
operations, I think it would be a terrific contribution and change
things dramatically in terms of the personal education we give ourselves
in those choices and in broader educational practice. On that, I'll
conclude. Thank you very much. \[APPLAUSE\]

Can I take a few, if you don't mind, before you go back to---

Reyna: I wanted to ask you a question.

Dr. Bjork: But, you know all the hard questions.

Reyna: Yes, I'm going to ask a couple of really hard questions. Suppose
the President of the United States or a member of Congress were to ask
you, "This is all very nice and good, but why should we take hard earned
taxpayers dollars and invest them in research on cognition and learning,
especially research that's based on work that's been done in a
laboratory. What practical benefit is that going to do for academic
achievement in the classroom?" What would you say?

Dr. Bjork: Well, I think you'd have to draw, in part, for that kind of
audience, you'd have to draw parallels. And, if you look at almost any
of the things in physics, physical, and engineering domains, where
things have changed our lives, transistors, lasers, and so on, and on
and on, those things did not come out of any applied research program.
They were basic research problems on how to make light coherent or how
the electrical property of nonregular kinds of substances.

And, I think it, this just prompted me to remember, I saw a while back,
they did some survey of what were the10 most promising developments that
would indicate we could make progress in battling cancer? Nine of
those10 came out of basic research laboratories as serendipitous kind of
findings. One out of the10 came out of the massive expenditure there's
been on cancer institutes and so on.

So, I would, in part, say, that you can look back on almost any field
and see that though there's a lag, it's basic research on understanding,
in this case, the very sophisticated, very unintuitive kind of
functional architecture of humans as learners or members, that will
provide the foundation for making real advances.

Now, the legitimate part of that question is when people look at my own
research, say, and, the kind of materials that we've tended to use,
they're very often a long way from real educational materials. Sometimes
as simple as meaningless paired associates.

And, so there is a genuine question about the generalizability, how you
take this basic research, how you test those implications, what does it
suggest for intervention or change in procedures.

But, basically, I think you could reverse the question as well. And,
say, to a large extent, intuition, face validity in standard practice
has been our guide for many years. And, are we happy with the outcome?
In particular, for those kind of populations that aren't highly select,
and where you're talking about learning that has some complexity. They
wouldn't stand still for as long an answer as that, but that would be
the shape of it.

Mr. : \[Inaudible.\]

Dr. Bjork: Yes, I think to become, there's a need, I even sort of, this
is something I think of as a question to myself, in a kind of selfish
way. There's a need to become what you could think of as sort of
meta-cognitively sophisticated. And, that doesn't just mean, that if
you're taking on the learning of a foreign language and so on, that you
would want to space practice and you'd want to exercise the production
retrieval, all the kinds of things you could say first, just managing
yourself. But, there ... more about your subjective experience. Don't be
fooled by being immediately correct on something that you could predict
or that was masked and so on. So, there would be a matter of trying to
learn what sort of subjective experience is a guide and is not.

And, in fact, part of that would have to say, it's tough for us. We'd
have to say, when I get that other outline on the industrial uses of
bacteria, what must have happened is that the process of making peace
between that outline and this article and kind of lining up, must have
created quite a sense of difficulty or temporary confusion.

What you'd like is the feeling, that the sense of learning is happening.
But, we think, we say, that's the sense of difficulty. We'd have to have
some more sophisticated notion of what learning is and what it feels
like. It's not supposed to be easy.

You probably heard the story about, this is probably aging me, CalTech,
the surely, you're joking, Mr. Fineman. Wonderful physics teacher at
CalTech. The Fineman affect. People have heard about that? Probably the
best intro. physics lecture ever. Fabulously entertaining at CalTech.
People flocking to take his classes. Somebody bothered to check on how
the people did from his class on the standardized intro physics exam
compared to other classes. Worse. Now, it's not that he was the worst
teacher. But, what happened, I think, is he made everything so
entertaining so apparently clear, he affected their motivation as to how
much they thought they had to study outside the class or on exams and so
on.

So, I've given a little more emphasis, maybe, than I should have, but I
think there is this kind of meta-cognitive part of this that is crucial.
Because learning isn't necessarily supposed to be easy or fluent. We
hear constant ads that always have the words of easy, effortless, and so
on in them for some kind of learning price or procedure.

Mr. : \[Inaudible for first part\] -- are we really ready to transfer to
a curriculum based experience that we can then test .... ... but we
ought to be teaching this way and we ought to do it right now, and then
test to see whether it works or not. Do you think we have... things like
that...?

Dr. Bjork: I think there's, you could take a given curriculum, at a
given stage, I think, and analyze carefully in terms of the kind of
components it ought to have. What sort of, what sort of things ought to
happen in the classroom, on the computer, outside the class, between
students and so on, to engage the kind of processes that basic research
says support understanding, retention, and transfer. So, I think it's
entirely possible. Is it easy? No it's not easy. And, I think it takes,
you know part of the difficulty in bridging the gap is, on the one hand,
to know about the relevant research, but then to have the kind of
knowledge of actual curricular experience materials in a given place, to
sort of know just what's the appropriate control group, and, then
basically, all those things involved to gain the permission, to do
things, that perhaps the students will go back and complain to their mom
and dad about.

So, I don't, I think it's a big, the potential is just simply enormous.
I mean, I may be too convinced, but I think we could, a variety of
results indicate that we could really upgrade the effectiveness of
instruction. But, all that's involved is substantial and shouldn't be
sort of underplayed. We shouldn't talk like it's easy. Back there.

Reyna : I should add something, too, about the perception of the proper
proposal. It's not that all the answers have been obtained. That would
continue to be a process and research ... knowledge is. The argument is
more that we have made a lot of progress that can be built on. Some of
the proposals will continue the development of fundamental knowledge in
concert with these ideas about educational significance. So, what we're
really talking about here is a new kind of research that blends
fundamental science with educationally relevant work, as well.

Dr. Bjork: I'll just add. That's a really good point because, also, even
something where the important thing we know is kind of empirical
principle, like the spacing of instruction facilitates long-term memory,
that has been a topic of research for a long time. There's 3 or 4 common
theories. None of those theories, actually capture all the different
spacing phenomenon. So, there remains a basic theoretical question
there, even though, on the empirical side, it couldn't be clearer.

Mr. : Will the notes and references from this talk be available on the
Web?

Dr. Bjork: I was told it would be transcribed, but your point about the
references is a good one. We can try to put those up, too.

Mr. : \[Inaudible\] \-- the question that I have is that, you know, is
the RFA about basic research, or is it about ... very different
tradition and with very different kinds of rules. The idea of design...
Ms. Resnick at the time was a researcher at MIT are very
different...development of robot design... I'd like you, I'd like for
you to talk about that issue. Because, as an issue-based researcher,
what can we do in the classroom, because... It takes a lot of years and
a lot of money to go from... talk about quantum mechanism...use quantum
mechanics in such a way that the power of it is...and not destructive.

Dr. Bjork: Yes, I'll let Valerie speak.

Dr. Reyna: Yeah, because, Professor Bjork actually did not participate
in the writing of the RFA at all. He was invited here for his many other
areas of expertise. That's a question I ponder about a lot, and it's a
very profound question. At this point, we did not limit it to merely
engineering, on purpose, so it was a very conscious decision. And, the
notion here is that the basic science and the engineering can go hand in
hand. At least in some proposals, the inference was that they would. So,
that there really is an research and development that goes back and
forth. And, I think there's some good examples in psychological science
of where that occurs successfully right now. One of those areas would be
the false memory area, in which practices in the courtroom are every day
changed as we develop the things that come out of laboratory tests, and
laboratory research is influenced by the needs of what happens in the
courtroom, and total areas of research are inspired by very practical
problems. I could mention several other areas, as well, but I think
that's probably a good one.

So, what I was taught in graduate school is to think of these things as
very different. There is applied, there's basic. But, I think what's
turned out to be the case in my own, you know, as through the trajectory
of my own career, and watching the careers of others, is that that
isn't, it doesn't quite work that way. That these things can proceed
together in tandem and influence one another and, in fact, some very,
very good work has come out of the results of that.

I will give you a concrete example of something. Okay, if you look at
theories of learning, theories of learning were hot a few decades ago.
And, there was a kind of diminution of interest in theories of learning.
They were not supported as basic science in some of the like the
National Science Foundation, the National Institutes of Health, the
Institute of Learning, and so on and so forth. And yet, I don't think we
can argue that, very few things are of such practical importance as
learning. Why? Because of schools, because of education.

If there had been an entity that was focused on what was relevant and
educationally significant, there is no way that we would have allowed
learning to fade as a popular area. We would have had a consistent and
stable basis of support. Because it's in the national interest to
support research on learning, scientific research on learning, so that
that base of work can flow to the classroom.

So, it's the practical realization that this is important that makes us
think about it as an important fundamental area. So, ... well, it's
educationally significant because that whole area of work is
educationally significant. And, it's with that in mind that we would
argue that it deserves some ... and stable funding. So, I think the 2
can go hand and hand. And, what I'm asking all of you to do is make the
case, make the argument. If you have a strong argument to make as a ...
make it in the proposal. We'll see what comes out of it. If people can
make an argument about fundamental science really being of significance
to academic achievement, and they want to do a fundamental science
proposal, and they can make a convincing argument, go for it. More power
to you. If you really feel you have to do something extremely applied,
that that's the only thing that's going to be relevant, make that
argument. And, get the reviewers to see the validity, to see the
purpose.

Dr. Bjork: You know, another comment, the example of witness memory
brings up another point. But, now that amounts to a human success story
for basic research on processes of identification, the way those go
wrong, and now we have a Department of Justice guideline that rests on
the work of cognitive psychology, but, what you can sometimes forget,
years went by where, in many places, I ran into this early on, you were
not permitted to even testify, let alone give your opinion. And, what
was the key thing? Well, something else happened. DNA testing. And,
somebody being able to show that something of over 90 percent of the
cases where people were falsely imprisoned rested solely or mostly on
witness testimony. And, I think, then, that says there can often be
other factors that can feed in and create a timeliness. I think, in this
domain of a kind of national commitment to upgrading education, some of
those other factors are kind of in place, I think, to create an
openness.

Sometimes it's a matter, I found out the hard way, in intervention,
sometimes you have to work a step up. I mentioned consulting at the
nuclear training center. And, when I would work with the individual
instructors there, they would get excited about things I would tell them
about. And, then next they'd say, "But, you know, I get evaluated and
promoted or not promoted based on what they do during training and the
expressions." Plus, the Nuclear Regulatory Commission, in some kind of
misguided way, they have a policy called negligent retention. You know
what that is? They record all the training, videotape everything for
these people that operate the nuclear plants. And, if you as a teacher,
so I'm going over something, and you, I ask a question and you give a
blatantly wrong answer, if I keep training, and obviously, if I'm going
to maximize training, not only should mistakes, people learn by making
and correcting mistakes, I would try to in some ways induce those and,
for what learning that could happen, I'd have them happen during
training, rather than operating the actual plant.

But, that's, it goes back to this business about innate talents and
having the gift or not the gift. So, if I'm a company in a training
program, I keep you in the training, after you've made this error on
tape, that's negligent retention. I kept you when you didn't have the
stuff to be---so, what happens? If you're a teacher and a student, what
happens? You try to structure it so nobody makes any mistakes, if you're
a teacher. If you're sitting out there as a student, if you're confused
and don't understand, do you raise your hand? Because you'll be on tape.

So, I mention that more than I want to. The basic point was, to
accomplish something there, it was necessary to work with the person who
was the director of training, not the teachers themselves. Because, they
were at too much of kind of a risk. So, training should be interweaved
in other things of that kind. It had to come at an administrative level,
one higher than the teacher trainer themselves. Yes.

Mr. : Actually, I think, as we anticipate bringing different experiments
and interventions into classrooms, what about our friends the teachers
who now have new emphasis on assessment and standards and therefore
might be less apt to try something that's different, because of new
constraints on them in preparing for assessment tests? Dr. Bjork: Yes,
the burdens that. I know Valerie made comments on that, too. I don't
know a great deal about all the emphasis that there's going to be on
assessment. But, one thing is it's not informed by a kind of value added
idea, recognizing that different teachers have different constituencies
among their students to start with. These kind of problems are going to
be exaggerated.

Reyna : You raise a very important issue. And, this is, with the new
emphasis on learning outcomes and adequate yearly progress, and these
types of things, are we going to discourage people from attempting to
innovate and experiment and to test things. And, of course, we have to
get the word out that simply because current practice, I think, as
Professor Bjork's lecture dramatically illustrates, current practice is
not necessarily safe. Meaning, that may not get you the best outcomes.
We can't assume, because that's the status quo, that that's giving us
the best learning outcome. So, if we take the scientist's view on
things, we get the question mark, is standard practice the best
practice? And, you know, in medicine, we have, often standard practice
is the control group. And, that would make perfect sense. What would you
do with students if you didn't do the new thing? Well, you'd do the
standard practice. What you might find out, if you randomly assigned, is
that your standard practice is better than what you innovated. You might
also find out, which is often the case in progress and science, that
what you're doing is worse than the alternative. You need to know that
quickly if you want to get results. So, in fact, I would argue, that you
must go out and get involved in research very, very soon. Get your
results right away. They don't really take, I mean, many people will
talk about randomized experiments as requiring thousands and thousands
of students. They really don't. Often, maybe a couple of hundred
students might be sufficient to get a very definitive result. And, you
would know right now, in the next couple of months, are you going in the
right way, or are you going to have trouble at the end of the year?

So, my argument would be exactly the opposite. What is really, really
important and you need to know, that's when you have to have research.
That's when it's urgent. Not when you want to just sort of, although,
again, I understand the emotional reaction that people may have. But, I
think we have to, again, think rationally about it.

Dr. Bjork: One kind of role, and I don't know if it's a role for OERI or
another division of the Department of Education. But, there are plenty
of success stories that, innovations, where there are long-term
measures, some of these have been Web-based interventions, some others.
And, really, in my experience, when I looked at those, it's been very
possible to see why they're likely to have engaged the kind of processes
that would create long-term learning, understanding, transfer. And I
don't know exactly how that would happen. But, in some way, the kind of
publicity in that there are different nonstandard ways of doing things,
and, that they're better in learning real science, math, whatever, could
be helpful to the whole body of researchers, basically.

Ms. : I have a question. We are looking for a partner in collaborations
and we see a vast number of people here. We want to know is OERI going
to provide a list of all the participants, and how soon that would
happen.

Reyna : That's a very good question. I was thinking about ways to do
that. You know, one of the things I was going to do, you know, really
create the opportunity to mingle. And, what part of the country are you
from?

Ms. : From the metropolitan area, right here in Silver Spring, Maryland.

Reyna: That's right. So, you're from Silver Spring, Maryland, and there
are other people who are from that area here. There was a question
earlier, and, again, I want to underline it, because somebody brought it
to my attention. Once again, this is a suggestion. It's not a mandate.
You know, government mandates have certain specialty status. But, this
is a suggestion. If you don't have someone on your team who's highly
familiar and especially, has published extensively in peer review
literature in cognition and student learning, you probably should get
one. Just because it's just too difficult, probably, to bring yourself
up on that whole literature in a short period of time. And, again, I
would turn that around and tell the folks who are laboratory scientists
to get partners who are in the schools who can talk about relevance and
significance in a deep and thoughtful way. Who live that. So, now that
I've encouraged you to become partners, the question is how can OERI
facilitate that. I've been on this job for about 5 months. I'm not sure
exactly all the things that are legal and proper, what we're allowed to
do by the rules. But, my colleagues and I are nodding at each other
right now. There's someway we can make the name available and so on.
Probably, the subject of confidentiality, the people who don't want
their names available, whether we can put that on the Web site or not,
we'd probably need to look into the consequences of that. But, I gather,
if people are interested, too, I would encourage you to contact, you
know, go to the journal. Look at things you're interested in and find
out who's publishing. Go to universities in your area, and look at
people in the different departments and see who might have a track
record of expertise. And, to the degree that we're able to help you,
we'll certainly facilitate that.

Mr. : \[Inaudible.\]

Reyna : Yeah, that's a very good suggestion. The suggestion was made
that we could post a directory and people could enter themselves into
the directory, if they chose to. That's an excellent suggestion.

Dr. Bjork: I have this little thing up here, just kind of wrapping up
the comment on the need also to adopt the learner's perspective. This is
a doctoral dissertation done by Newton at Stanford. And, she divided the
participants into tappers and listeners. And, the tappers looked at a
list of paper with the name of 25 simple melodies and they tapped
out---\[makes tapping sounds\]. And, then the listeners didn't have that
list, and they tried to say what the melody was. She then asked all the
tappers to try to predict the likelihood the listeners would be correct,
and across all tappers, just about 50 percent, they predicted about half
the time the listeners would identify the melody. The listeners' actual
performance was shown there. What did I tap out, incidentally? What?
Row, row, row your boat, gently. What happened in my head, music,
lyrics, orchestration. What did you get? \[Tapping sound.\] That's a
fundamental problem we face as teachers. Piaget was quoted as saying
that almost every beginning teacher is prone to be talking to himself
until he sort of learns, you know, mindful of only his own point of
view.

And, the last thing I'll show, there's always wisdom in Calvin and
Hobbes. There's a great tragedy that he's not doing that. You know we
somehow expect our students to remain efficient for a 50 minute lecture
or day long, in schools. Now, 75 minutes in college and stuff. I tried
to remember how hard it was to listen to the preacher talk for about 20
minutes. But, as those of you who are Calvin and Hobbes fans know, he's
not too keen about school. \[shows cartoon overhead\] So, wonder how
long it's been since I last looked at the clock. Maybe it's been an
hour. Well, actually, it's probably been only 40 minutes. I guess, half
an hour, to be safe. \[Laughter.\] It's going to be a very bad day,
Calvin. \[Laughter.\] That's it for me. Thanks for remaining alert.

Office of Educational Research and Improvement (OERI)

***Cognition and Student Learning (CASL) Research Grant Program***

Pre-Application Meeting

February 19, 2002, Washington DC

**Bridging the Science of Cognition to Educational Research and
Practice**

Robert A. Bjork

*University of California, Los Angeles*

**Overheads:**

1.  Manipulations that introduce desirable difficulties for the learner;

2.  Bahrick (1979): spacing effects in vocabulary learning;

Shea and Morgan (1979): blocked versus random practice;

3.  S&M: Apparatus;

4.  S&M: Results

5.  Lemire and Ste-Marie (1996): blocked vs. random printing;

6.  Kerr & Booth (1978): Variation;

Mannes & Kintsch (1987): Comprehension of text;

7.  M&K: Overall design and sequence of events;

8.  M&K:Article to be studied, page 1;

9.  M&K:Example of consistent outline;

10. M&K:Example of inconsistent outline;

McNamara, Kintsch, Songer, and Kintsch (1996):

11. MKS&K: Heart disease---fully coherent version;

12. MKS&K:Heart disease---minimally coherent version;

13. MKS&K:Text recall: High and Low Knowledge Groups;

14. MKS&K:Problem solving and bridging inference questions, both groups;

15. Baddeley & Longman (1978): Mean satisfaction with training;

16. Simon & Bjork (2001): Metacognition in motor learning;

Benjamin, Bjork, and Schwartz (1998): Mis-measure of Memory;

17. BB&S: Method;

18. BB&S: Results

19. List of conditions of instruction that can mislead the learner

20. Shepard's (1990) table illusion: From "*Mindsights*" book.

**References:**

Baddeley, A.D., & Longman, D.J.A. (1978). The influence of length and
frequency of training session on the rate of learning to type.
*Ergonomics, 21,* 627-635.

Bahrick, H.P. (1979). Maintenance of knowledge: Questions about memory
we forgot to ask. *Journal of Experimental Psychology: General, 108,*
296-308.

Benjamin, A. S., Bjork, R. A., & Schwartz, B. L. (1998). The mismeasure
of memory: When retrieval fluency is misleading as a metamnemonic index.
*Journal of Experimental Psychology: General, 127,* 55-68*.*

Bjork, R. A. (1999). Assessing our own competence: Heuristics and
illusions. In D. Gopher and A. Koriat (Eds*.), Attention and peformance
XVII. Cognitive regulation of performance: Interaction of theory and
application* (pp. 435-459). Cambridge, MA: MIT Press.

Bjork, R.A. (1994). Memory and metamemory considerations in the training
of human beings. In J. Metcalfe and A. Shimamura (Eds.),
*Metacognition:* *Knowing about knowing* (pp.185-205). Cambridge, MA:
MIT Press.

Christina, R. W., & Bjork, R. A. (1991). Optimizing long-term retention
and transfer. In D. Druckman & R. A. Bjork (Eds.), *In the mind's eye:
Enhancing human performance* (pp. 23-56). Washington, DC: National
Academy Press.

Jacoby, L. L., Bjork, R. A., & Kelley, C. M. (1994). Illusions of
comprehensions and competence. In D. Druckman and R. A. Bjork (Eds.),
*Learning, remembering, believing: Enhancing human performance.*
(pp.57-80) Washington, DC: National Academy Press.

Mannes S.M., & Kintsch, W. (1987). Knowledge organization and text
organization. *Cognition and Instruction, 4,* 91-115.

McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are
good texts always better? Text coherence, background knowledge, and
levels of understanding in learning from text. *Cognition and
Instruction, 14,* 1-43.

Schmidt, R. A., & Bjork, R. A. (1992). New conceptualizations of
practice: Common principles in three paradigms suggest new concepts for
training. *Psychological Science, 3,* 207-217.

Shea, J.B., & Morgan, R.L. (1979) Contextual interference effects on the
acquisition, retention, and transfer of a motor skill. *Journal of
Experimental Psychology: Human Learning and Memory, 5,* 179-187.

Shepard, R. N. (1990). *Mind sights: Original visual illusion,
ambiguities, and other anomalies, with a commentary on the play of mind
in perception* (p. 48)*.* New York: W. H. Freeman.

Simon, D. A., & Bjork, R. A. (2001). Metacognition in motor learning.
*Journal of Experimental Psychology: Learning, Memory, and Cognition,
27,* 907-912*.*

Ste-Marie, D. M., & Lemire, J. A. (1997). *Contextual Interference: A
Strategy for Handwriting Skill Acquisition?* Unpublished paper.