Archived Information

**Cognition and Student Learning Research Grant Program**

**Proceedings from the Pre-Application Meeting of**

**February 19, 2002**

**Washington, D.C.**

**New Directions in Educational Science**

**Valerie Reyna**

All right. I'm going to give you a little bit of background. I'm going
to talk a little bit briefly about the context for this competition. As
I sort of alluded in my introductory remarks, this is really a new OERI.
Now, there have been people toiling in the wilderness in the past, doing
things that are more in the vein of what we're going to be talking
about. And, to those people we say thank you. Because, they've been
doing scientific research all along. They've been doing some good
things. Some of them have been funded by OERI. Yes.

However, I think we can make the argument now that the combination of
conditions that have led to this kind of initiative is unprecedented.
The conditions that are operating now that make this possible are
different than at any other previous point. And, why do I say that?
Well, we have legislation that talks about scientifically based research
undergirding practice. And, it's mentioned over a hundred and eleven
times in that legislation. And, that sort of thing has not occurred
before.

Now, we could quibble about the definition of scientific research, and
some people will. But, frankly, I think we ought to see this as a
positive symbol and think about this in a positive way. Yes, you might
talk about this and might talk about that around the edges, but the main
issue, the message here is extremely important. That high scientific
standards of rigor are going to be applied to what's done with students
in the classroom. And, that we can look forward to, ultimately, the
practice of education being evidence-based. And, of course, that's where
you come in and where research comes in. It is pivotal; it's
foundational. Because where does the evidence come from? It comes from
scientific research.

This is unprecedented. We have unprecedented leadership. Russ is gone,
so we can talk about him. But, I'm not going to say anything different
than I said when he was here. He is a bona fide scholar and a scientist.
He's actually done research. He's done the sorts of things we're talking
about. So, that level of leadership in that particular position is
unprecedented.

We have support for research that goes, you know, up to the White House.
There was a summit on cognitive development. How many people know about
the summit on cognitive development that was at the White House? Not
everybody. You know who most of the speakers were? Researchers, talking
about actual research on cognitive development. So, the President, the
First Lady, folks like that, think that this is important. People in
Congress think this is important. This has broad bipartisan support. If
you look at the definition of scientific research in the legislation,
it's quite appropriate. It's technically correct. It has all the commas
in there. It's very impressive.

And, I think, this really does begin, as I alluded to, with people.
People are so important to an initiative such as this. Not only people
with good research credentials, and that's extremely important. But,
people who feel a sense of urgency about what happens with students.
People who realize that, right now, there are students who are going
down for the third time, who are being lost. And, that there's a better
way to do things. And, there's a better way to make progress and to
build on the successes that, perhaps, are isolated. And, the scientific
method is one way to do that.

Now, is science the only way to do anything useful in the world? We're
not saying that. We're really not saying that. I swear, we're not saying
that. It's not the only valuable thing in the world. However, it is a
necessary component of practice. Not a sufficient one, but a necessary
one.

So, people will make decisions on the basis of values. And, that's not
about scientific evidence. But, shouldn't the facts be part of that
decisionmaking? And, again, this is where you guys come in, so thank
you.

There's a number of ongoing initiatives. And, I'm going to briefly touch
on those, and then I'm going to get right to this initiative. But, I
thought you might want to, since you came all the way to Washington,
some of you, hear about some of the things that we're doing.

There is IERI. We have one of the representatives, Mark \[Constas\],
over there, if you want to ask detailed questions. He's the source for
that. That's an interagency initiative that is nicely funded and is in
collaboration with NSF and NICHD, as well. So, that's a joint enterprise
that's been going on for some time.

There's a National Language Minority Panel that we're putting together
right now, that I've been very involved in. And, given the demographic
changes in this country, I think there's a real opportunity. There isn't
a whole lot of research, probably, that's very high quality research in
that area yet. There's some. And, there's a need at this critical
juncture to summarize the good material and to set an agenda for the
future, so that we can build on that. So, we don't have to say that
we're just speculating, and wouldn't it be nice if we knew the facts. We
can move forward in that extremely important area.

A lot of people are talking about character education and prosocial
behavior and the Freedom Corps and things of that nature. So, when we
talk about science, we're not talking about a heartless and soulless
science. We're talking about a science that bears on things that are
important to us as a nation. And, character certainly is important to us
as a nation, and character and prosocial behavior, in schools, are
important. Because it's important to have an atmosphere in which
students can learn. And, students can't learn in an atmosphere of
violence and where their neighbor is interfering with their learning.

So, those things are important as well. We have a math report (on the
status of mathematics research) that was commissioned and being
conducted by the Rand folks. We hope to get a report soon. It's a bit
overdue, but we're waiting with bated breath. I think some day people
will look at mathematics the way they now look at reading. And, people
have talked about reading as a civil right. Because it opens the door to
so many other things. Well, in a modern, technological society,
mathematics can be viewed very similarly.

If we think about the elementary years, the ratio concept (fractions,
and the sort) is a barrier to success for many students. And, I think
that's well documented.

You move into the middle years and you're talking about algebra being a
tremendous Waterloo for students and a barrier to success. And, you look
at high school and college, and what is it? Calculus. Right. That's the
big Waterloo. And, people change majors and change the course of their
lives based on algebra and based on calculus. I know people, people in
my family, for example, that dropped out of school when they got to
algebra. It was just too much and they left the eighth grade and said,
I'm not going back there.

So, these things are extremely important gateways to future success.
And, it's not that everyone should be a mathematician or a scientist.
We're not saying that. What we're saying is that you shouldn't have a
choice forced upon you because you didn't have the right teaching.
Perhaps you had the aptitude, and you would have been successful and you
could have had a choice of careers that were your preference. If you
were able to master these counter-intuitive things. And, they are
counter-intuitive.

So, the freedom and the opportunity to make that choice have to be made
available to students. So, I think, I hope, that mathematics is going to
be looked at, at some point, as a civil right.

We're talking about comprehensive school reform and how to get more out
of that portfolio of funded research. How to draw across the studies
that have been done and do something that is more coherent and
systematic. And, we're working on that. You probably have visited our
Web site, so you know about the preschool initiatives that we've got.
We're very interested in early, you know, preschool readiness to learn.
The What Works Clearinghouse is something we're working on, too, you may
have seen something about that. So, we can make the products of research
available to practitioners and to policymakers is an easily available
format to assist decisionmaking.

What about cognition and student learning, which is why we're here? You
know, that was just a very quick preamble for you. We can talk about
really two main axes that your grants will be assessed on: scientific
merit and educational significance. Now, scientific merit involves
methods and measures (are they reliable and valid), and so on and so
forth. Significance or relevance, on the other hand, is the item on
which we're really getting a lot of questions. And, that makes perfect
sense. Significance and relevance are very profound issues. This is not
as straightforward as scientific merit. It really is a question of
judgment. And, we've gotten questions such as, how directly linked
should this be to student outcomes? Should this be in the classroom
physically or otherwise, it's just not relevant? Can you talk about
learning through laboratory tasks and can that be educationally
significant? Or, are you just talking about neuroscientists working in a
laboratory, maybe with rats? Is that somehow educationally relevant?
What do you mean by that?

So, let me try to tell you what I think is meant by that. And, later on,
we have an expert. Professor Bjork is going to talk about that. And, I
encourage you to stay for that lecture. That will be very interesting.

Ultimately, though, we have to warn you. The remarks I'm about to give
you are simply my judgment. The actual judgment that counts will be made
by the reviewers. So, what I'm going to give you is a series of
suggestions, based upon my experience, which is quite extensive, with
lots of different kinds of reviewers. Please feel free to ignore my
remarks. Feel free ignore Professor Bjork's remarks. It's ultimately
your decision as to what you think is educationally relevant and
significant, and you need to make the case in your application. But,
here are a few tips that have been helpful to me over the years.

You know about the four criteria, and significance is one of them. There
are some rules of thumb that people use, for example, in medicine. They
look at what's called the burden of illness. Have you ever heard that
expression, the burden of illness? They look at things like the number
of people affected by a problem. All other things being equal, if a
problem affects more people, as opposed to fewer people, it goes up that
significance scale. If it affects two and a half people---you know, all
other factors being equal...

What do we mean by the severity of the problem? Well, this is, in
medicine, this is the burden of suffering, for example. In education, it
would be something like that. How severe a problem is this? I tried to
make an argument that mathematics is a severe problem, because it
prevented people from fulfilling their potential. When you look at
income that people make, it's related often to their major, which, in
turn is related to their number of math courses or whether they were
successful in those courses. So, I think that that's a severe problem.
Because it makes choices for you that you don't want to make. You don't
have freedom of choice; you don't have the opportunity to be successful
in the economy.

So, those kinds of arguments, again, that's not a strictly numerical
argument. However, I would encourage you to use facts in this section.
If you've got statistics you can bring to bear, talk about that. You
know, again, when I was doing research in medicine, I would make the
following kind of argument. And, I'm picking this because it's medicine,
so I'm not picking on any one particular topic, so I'm not giving a
signal of what I think is significant. You need to make the argument
about what you think is significant.

Well, I would say something like (I was interested in medical
decisionmaking under uncertainty), so, I said, well, 6 million patients
come to the emergency room every year complaining of chest pain. And,
20,000 of them are discharged inappropriately, go home, and die. This is
an important problem. I would argue that's a significant problem,
therefore. All these patients dying, what do you do about that? How do
you make this difficult decision about chest pain?

So, we've included in your handouts some handy dandy statistics. This is
just a first pass. They're not even, in my opinion, sufficiently
digested to really give you some guidelines about what would point to
significant problems. They're just the beginning of statistics with some
citations or sources for more. This is from our National Center for
Education Statistics. I would encourage you to go out and get more
in-depth statistics, even than that. Again, that's supposed to be kind
of an appetizer there, not the whole meal.

So, bring to bear as many arguments as you can. Get right into that
right away. As Ann mentioned, you really don't have a lot of space in
this application. So, boil it down, boil it down, get right to it. And,
make your arguments in a very substantive way.

We avoided saying, although the Assistant Secretary mentioned the
classroom, clearly, if you talk explicitly about school contexts, and
explicit school related outcomes, or learning related outcomes, where
the reviewers don't have to connect any dots, and the reviewers don't
have to make any inferences, that this is relevance. Because, it's
obviously, superficially, at that level, literally relevant. Clearly,
that makes sense. So, if you can take your measure and at least relate
it to a school outcome, a school context, in at least part of our
proposal, then there's no question of having to infer that it's
relevant. Obviously, reviewers will usually find that more compelling.
They don't have to connect dots. They don't have to look between the
lines for you. If you have your reviewers looking between the lines for
you, usually you can't count on that. So give them ammunition, if they
like your proposal, so, they can say, yes, see, it says right here that
this is correlated with---and then you give some actual academic
outcome.

Now, what if you haven't got that? What if you have been doing
laboratory tests? Well, maybe you go out and get some pilot data right
away that connects what you've been working on in attention, reasoning,
and memory, to an actual academic outcome. So you can present that as
preliminary data. It hasn't been published yet. It's just pilot data.
But, it shows a significant correlation, or it demonstrates a
significant difference. Then it's not a theoretical argument that it's
relevant. You've shown that it's relevant.

Or, another strategy that some people have used is the transitive
reasoning strategy. It goes something like this. You've been looking at
a process, say, in attention. It is correlated, say, with a
neuropsychological outcome on a test. On a clinical test that is widely
used, that's valid and reliable. That clinical test has been related to
these other academic outcomes by other people in other studies, which
you now cite and discuss briefly. So, see, the transitivity argument?
Therefore, your processes are related to the academic outcomes. Because
the thing in the middle is related to both of them. You see? So, you've
made the case that this is, in fact, relevant. You haven't sort of just
exhorted the reviewers, doesn't this sound relevant? You've made an
argument with evidence that it's relevant. I would encourage you to do
that wherever you can.

Have very precise hypotheses that can be falsified. This is the
scientific method; have hypotheses that are tested in the proposal. If
you know someone who has a long track record of successful NIH or NSF
proposals, talk to them. There will be very similar kinds of criteria
applied, in the general sense.

If you, yourself, have not published extensively in cognition and
learning peer reviewed journals, it's going to be very difficult for you
to build knowledgeably on the work that's printed in those journals.
But, do not despair. This is where collaborations come in. Members of a
team are not expected to have the same background. Otherwise, it
wouldn't be a very useful team. You don't have nine pitchers in
baseball, right? You have a pitcher and a catcher and, and so forth;
they bring different strengths to the table.

However, if you have no one on your team who is deeply familiar with the
peer reviewed research in this area, that would probably be, it would be
difficult, then, to premise your application on that literature. So, I
would encourage you, at the earliest opportunity, to get that kind of
technical help.

On the other hand, there are many people in the audience here who are on
the front lines of significance. They're in the schools. I would
encourage those of you who have been in the laboratory to talk to the
people in the schools and ask them what they think is significant, and
listen very carefully to what they tell you, and partner with them.
Because they bring strengths to the table.

So, I think you know where I'm going here. We really are talking about
bridging these two domains.

All right. So, another rule of thumb, take it for what it's worth, is do
not try to solve all of humanity's problems in a single grant proposal.
Significance doesn't mean unfocused. It doesn't mean you're going to be
everything to all people, all areas. We listed areas, but we don't
expect most applications to cover all those different areas. It would be
very hard to be an expert in reasoning and memory and attention, and all
those other things, and cover all the subareas. So, focus, focus, focus.

Describe a series of programmatic experiments or studies. And, by that,
we mean things that build on one another. The first study doesn't answer
every question. It leaves open some interesting questions. The most
important one it leaves open, that's study number two. That leads to a
fork in the road. And, that's experiment three and four, you see. And,
so you build in a sort of coherent story line, from beginning to end.
So, you want to do in-depth studies.

And, here's a very, very, difficult and subtle point. What about, if you
really feel that what you're doing is, say, theoretically big, and by
that, we mean empirically tested and supported theory. Not just, "I
speculate." But, an empirically supported theory that makes predictions
and explains learning phenomenon. What if you want to study something
that isn't situated in a classroom, that you really feel is
educationally significant? So, say, for example, you had a learning
theory (I'm making this up obviously), that you thought would increase
the learning rate by 50 percent. And, you had some pilot data on the
acquisition of second language learning. And, you showed that, based on
the mechanisms in detail stipulated by your model, you got a 50 percent
increase in learning rate. Would that be educationally significant? I
think it would be. It would be very cost effective. And, you might want
to explore the boundaries of that.

Again, ideally, put an educational context in there somewhere. And, your
reviewers will not have to connect the dots for you. But, we
specifically are not mandating that it has to be in an educational
context, because it's possible that you could make the case that you're
educationally relevant, based on the nature of what you're doing. The
questions you're asking and the answers you anticipate getting. So,
we're leaving that open, so that you can make the case that you're doing
some fundamental work that has to do with the building blocks of
learning theory. You could compete under this. We'll see how it goes.
We'll see how the reviewers react anyway.

What about which student populations? Now, again, we specifically left
open the student populations. Some people have called and said I want to
study preschool. Other people have said I want to study adult learning.
Some people have said, I want to study people who have ADHD. Other
people have said, I want to---go forth. The only argument you have to
make, remember again, is that this is significant. So, if it's
significant and you can make the argument that this is an important
population to study because... then it certainly would fall under our
purview. We're going to try to fund the best work.

There's been a lot of discussion about methods and you saw in this
application, there was a list of methods that we talked about. We were
very explicit about that. Define your variables. It will help the
reviewers a lot. We're really doing you a favor. I've sat in many
competitions where we couldn't understand the proposal. They kind of
liked it. There was one I did recently myself, where I really liked it,
but I wasn't quite sure what they were doing, but I kind of liked it.
Now, if they had been clear about what their independent and dependent
variables were, the predictor and criterion variables were, if they had
been clear, it would have done better. Because it sounded like there
were good ideas in there, if only we could be sure what they were. So,
the message here is that we encourage people to be very precise. Exactly
what are you doing? How many subjects in experiment two? What exactly
are the variables? What exactly are the factors? You should just be
explicit about all of that. Don't just do hand waving.

And, again, something that may, and has already raised questions is what
about this randomized experimental design, and is that really the best
design? You've heard it here. We are not saying that that's the only
design that is a scientific design. Of course not. The National Academy
of Sciences recently released a report, if people want to read about
that sort of thing, in which they made a similar point. What we are
saying is that your question should fit your design. If you're asking a
cause and effect question, does this work or not, then, yes, the best
design to answer that particular kind of question is a randomized
clinical trial or a randomized experiment. Random assignment, not random
selection.

So, on the other hand, if you're asking a different question, you might
pick a different methodology. But, be very clear that the question and
the method bear on one another.

Another issue that we talk about in the FRN that I want to underline is
to definitely talk about the prior literature. Science is cumulative. It
builds on the discovery and work of other people. And, I don't mean
prior pontification. If people have pontificated and speculated and said
their opinion in 90 articles, don't worry about that. That's not
evidence. It might be nice; it might be interesting. If you want to cite
it, fine. But, what I'm talking about is evidence, prior evidence. It
will be viewed dimly by reviewers if you have not cited the relevant
prior work in an area if you're proposing new work in that area. You
should, in fact, not just cite it. You should talk about the strengths
and the weaknesses in that prior work. "Well, the conclusion section
says this is true, so I'm citing it as gospel." Not good enough, right?
If there was some fatal flaw in the method, you can't just cite it
uncritically. You cite it evaluatively. I think a lot of people probably
already know that.

So, in closing, in deference to our extremely organized master of
ceremonies over there, this really is an unparalleled opportunity to do
something important. You really are helping your country. We want to
create a new tradition, which will become routine---that practice in the
classroom is evidence based. You are the vanguard of that revolution.
People will look at this: these applications, the quality of the
applications, the arguments you make about significance will be looked
at very carefully. Some of the people who look at them will be
skeptical. So, be convincing. I know you can be. And, some people will
wonder, are there that many people out there who have something to offer
at this point in history to educational practice. I'm betting that you
do. And, some day, when the practice of education becomes routinely
evidence-based, I want you to look back and I want you to remember this
day, and I want you to say to yourself, I was there, and I helped make
it happen. Thank you.