Archived Information

# Cognition and Student Learning Research Grant Program

**Proceedings from the Pre-Application Meeting of**

**February 19^th^, 2002**

**Washington, D.C.**

Following is a transcript of a question-and-answer session at that
pre-application meeting for this grant program. Participating for the
Office of Educational Research and Improvement, U.S. Department of
Education: Dr. Valerie Reyna, Dr. Ann Mullen, and Pat Knight.

**Questions and Answers**

Q: The question is, nothing is supposed to be in the appendices that's
critical to the proposal, but what about actual survey test items? Do
you want copies of the actual test included in the appendices?

REYNA: It certainly doesn't hurt to have the actual test itself; it
eliminates certain uncertainty. Again, you're absolutely right, and I
need to reiterate, the application should stand on its own. You
shouldn't have to go to the appendix in order to comprehend something in
the application. So, it should be described and characterized and a lot
of specifics given about it. But, I think it is helpful to have the
actual item.

Q: And, this would mean the appendices would have to be sent in with the
hard package.

REYNA: Yes, and there's a limit on the number of pages in the appendix,
too. Some of these things would be longer than that, too. That's another
issue. You're limited to, what, 20 pages in the appendix. You may want
to do selected items that are most relevant to the kinds of things in
your proposal, rather than the whole set. So, here's an example of this
type of item, this sub-scale. Here's an example of another sub-scale,
that sort of thing. You know, often having a concrete example just
dispels a lot of misconceptions that the reviewer might have about the
item. Now, ideally, we're going to be looking for reviewers that have a
lot of technical competence in these areas. But, even so, a concrete
example is very, very helpful. But, obviously, it couldn't be exhaustive
at 20 pages.

Q: Is there a deadline for asking questions?

REYNA: You can continue to ask questions through April 19th. And, we
will answer them, to the best of our ability. Absolutely.

Q: If there is more than one institution. You can do that by having a
subcontract or by having each institution make an application. Would you
have advice or guidelines on how to do that?

MULLEN: That's probably a good question for our technical experts over
here---do you have recommendations on that question?

KNIGHT: You have to have a lead applicant, and that is who will own the
grant. We have only one relationship with one. It's probably good then
to establish some kind of collaborative agreement within your group so
you know who's going to do what, outline that however you want. You
could do it through the contract or you could just do it through a
collaborative agreement. It would be up to you. But, it should be
detailed in your application who's doing what and who's the grantee.
But, the main thing, there's only one grantee.

REYNA: On the science part of that, let me add, collaborations across
institutions, especially in the modern Internet era, are becoming more
and more common. We would encourage that. It does get complicated,
primarily due to your institutional rules about subcontracting and that
sort of thing. But, there's no reason to think that people could not do
collaboratives across institutions. And, the key, of course, is that
there really has to be a rationale for that. A scientific rationale, and
not just more is better. But, that it actually fits together in some
coherent way.

Q: Maybe I missed it here, but I don't see a reference to a signature
endorsement or institutional endorsement, like the signature from the
director of sponsored research. How do you submit that in the
E-application?

MULLEN: If you do the E-application, there is one form that you will
need signatures on. And, that form will be faxed in. And, all the
instructions for that are on the E-application site. It's the cover page
form. I think it's the first one is listed on that list of forms.

Q: Are there any incentives for collaborating?

REYNA: No, there are no incentives. And, in fact, collaborations are
complicated. So you really would have to make a scientific merit
argument here. It would have to be they're the key expertise or this
person has, obviously, the best and strongest arguments for reviewers in
my experience is "this is the best person in the world," who's at
another institution. "They're the person who created this survey." Or,
they're the person who's the leading expert and published extensively on
a particular analytical approach. Or that sort of thing, so you bring
them in and you make the argument on expertise. Now, if you don't have
the expertise in house, the incentive is that you're more likely to be
funded. Because, you can make an argument that this is a better
proposal. But, beyond that, there's no special incentive.

Q: The question is the specificity of the research design and analysis.
Question is, will the reviewers be trained to look for that?

REYNA: I'm hoping, frankly, that if we had to train someone, they
wouldn't be selected as a reviewer to look for that. We're going to try
to get people who are themselves models of good science and good
research, and, in their own work, embody the kinds of criteria that we
set forth. So, we're going to be looking for experienced reviewers who
would know how to do this.

Q: \[Inaudible.\]

REYNA: They will get instructions about our criteria and exactly how to
weigh them so that all the---because, you know, if you get a very
experienced reviewer, say, you've reviewed a lot for NIH, they have a
slightly different criteria and NSF has slightly different criteria, so,
yes, they would receive instructions about how to apply those. Yes,
absolutely.

Q: What do you do with the letter of intent, at this point?

MULLEN: Those are for information purposes for us to have a sense of who
our applicants would be and the topics they're considering. So, it's
just a way for us to gather and collect information.

Q: My question is, if you have a leadership team that consists of four
to five people \[inaudible\]?

REYNA: The question was, if I understood it, that, a junior person,
perhaps, who doesn't have a long track record of publication would
collaborate with a more senior experienced person, would that improve
the applicant's chance of being funded? Yes, in my experience reviewing,
and, again, remember the ultimate decision is the reviewers' decision,
but based on my long experience with many different contexts, I would
say that, almost always, people will look at the fact there's someone
experienced, who has successfully accomplished what this person is
proposing to do, and is available as a resource, and that would be
considered an asset. And, I think it's probably a good strategy.

Q: On budget availability: I'm looking at your estimates of funds
available. Are you talking about total cost or direct costs? And, two,
is this the annual budget that might go on for 3 years, or is this one
budget that's the total for 3 years?

MULLEN: No, that is the total cost. It includes both direct and indirect
costs. All the figures specified are the total amounts and include the
direct and indirect costs. And, that is the entire budget for each year
of the 3 years. It's 3 million a year for each of the 3 years. The
average amounts are listed in the Federal Register Notice and they range
from 75,000 to 500,000 per grant per year. And, the amounts listed that
are the amounts per grant are per-year amounts.

Q: The question is, do they get all the money the first year, or do they
get money the first year and then money for a second year and then money
for a third year? If the budget for the project period is 3 years. What,
are there three separate budget periods?

MULLEN: The budget is written for each year of the project. If the
project continues successfully, they will receive the money for the
first year of the budget in the first year, the money for the second
year of the budget in the second year, and so on.

REYNA: If people want more help, too, on the budget kinds of things, we
can follow up this meeting, too, with an individual. We'd be happy to do
that and make sure people have specific answers to that.

Q: Are you anticipating another competition?

REYNA: Yes. We are anticipating another competition that would be a
Cognition and Student Learning Competition. And, the amount of money
that we anticipate, this is the government, you know, so you can't say
for sure. But, we strongly anticipate that the amount of the money is
going to be more the next go round. But, this is for this round of
competition. So, this is the beginning. And, in fact, one of the
arguments for support for money in this area is the number and strength
of applications. That's where you all come in. So, if people send in a
lot of good applications and say they really could do more and this is
what could be done, then that's one of the ways the government says,
gee, there's a supply issue out there. There really is depth, there's a
capacity in the field. So, in fact, perhaps we ought to fund this some
more.

Q: \[Inaudible; still background talking\] -- From the Federal Register
Notice, it looks like you can go the route of either choosing cognitive
psychology or neuroscience as your basic research framework. It doesn't
sound like you want both.

REYNA: You can choose. And you can either name cognitive or
neuroscience, if you wish, for those people in the audience going wait a
minute, you know. I'm both, oh, no. You can pick, and you can pick an
area within that. And, within attention, you can focus on selective
attention or response inhibition.

Q: So, if you choose to go purely in the cognitive process---

REYNA: That's perfectly okay.

Q: The other question is that, in the cognitive processes, you broke it
down into one and two. One being information processes, two being higher
order cognition. And, I'm not sure that we at the University of Central
Florida, we'd like to do a little of both.

REYNA: Right. Of course, when you, like, write a text book, when you
have these artificial chapters, and one of them is called this, you
know, basic information processing. And, another, part two of your book,
is called higher order cognition. The reality is that we're lying to our
students, of course. There really is no definite break there. So, that's
just a question of convenience, of trying to get people to think very
precisely about what they're doing. Now, I was not advocating, again,
these are all suggestions. I'm not giving you a mandate. My suggestion
is try to be precise, though, about what mechanisms are being affected.
If you're saying there's a whole series of information processing
stages, that they feed back on one another, be real explicit about that
and explicit about what you're talking about. Just sort of generally
something about learning is probably not precise enough, or something
that's so global that it attempts to cover the universe, but you don't
really get into what's the mechanism, what's the hypothesis of the
action going on here. So, those are just, think of those items as sort
of just retrieval cues. As little points of departure on which you can
do a much better job than we've done.

Q: \[Inaudible.\]

REYNA: I agree. Again, this is another call for collaboration. And, I
have been trying to say something along those lines during today's
events. I think it's, people who have thought for 20, 30 years about
what's educationally relevant are likely to have more profound and
detailed notions about what really is relevant in the classroom, people
who've been on the front line. They have an in-depth perspective that
someone who hasn't thought about it for 20, 30 years and hasn't
experienced that, is unlikely to have. Similarly, it works the other
way. So, again, I agree with you. Working, by building on current
collaborations and current partnerships, that's a strength that you
bring to this proposal. I would encourage that.

Q: To what extent would you be interested in seeing research, basic
research, that relates to new media, like the World Wide Web or
television. Also, what kinds of dissemination of the results should
funding proposals discuss, or how will the program overall utilize the
results to get them to have some effect.

REYNA: Well, first question is what about technology and so on. Again,
technology, for its own sake, we would not be interested in. Technology
in the service of cognition of learning and academic achievement, we
would be very interested in that. And, then the argument becomes one of
scientific merit and significance. It's exactly the same argument. I
think, I can say, as someone who headed a division of learning,
technology, and assessment, that there's sometimes a little bit too much
faith in technology, that technology will somehow take care of the
problems of learning just because it's, well, expensive, and it has some
nice bells and whistles associated with it. But, you know, again, as
Professor Bjork's talk illustrated, that's not necessarily learning.
That doesn't necessarily, in itself, foster learning. So, the argument
would have to be a very high level scientific argument with hypotheses
about learning mechanisms involved. And, yes, indeed, if technology, for
example, could allow you to deliver something to many, many more
students over the World Wide Web, you could affect millions, as opposed
to scores of students. That seems to me an argument about significance,
for example. So, it would depend on how you wove it into the proposal.

Your second point had to do with dissemination. And, my promise to you
is that, if there are, this really is the first initial competition in
this area, it will be a model for future endeavors of OERI. If we get
some good, successful proposals that we think are paragons of what we
want to encourage, we will disseminate that, you know, far and wide, we
can promise you. And, we'll be very interested in doing that. And, I
think, you know, as I said to the Secretary in a conversation, Secretary
Paige, it's really more of a question, more than dissemination. It's
really a question of communication. Not just talking at people, but
communicating with people about their needs and talking about how this
fundamental work can address those needs and fit into those needs and
help the students, ultimately, in a very real way.

So, believe me, we'll be on top of the Hill and talking about the work
and so on. Just send us good work.

Q: Are all potential applications weighted equally, or might there be a
sense that some agencies in the government are funding and already
researching some domains. For instance, NICHD is really all over reading
and biological aspects of reading. NSF and IDRI are really pushing, you
know, science education. And, some domains are, there's nobody going
after them.

REYNA: I think that's what you put in your background and significance
section. You have to make an argument in any new research that you're
solving a problem that hasn't already been solved that's a really
important problem. If it's a problem where it's a question of tying up a
few loose ends, scientifically, it's not going to impress reviewers, in
my experience, as much as you're saying we're going to break new ground.
But, believe me, in my review of the literature, there's so much that we
don't know about science and mathematical learning. There's so much we
don't know about reading. There are many, many, so, I wouldn't take a
domain, and just rule out a whole domain. I think, again, you have to
make an argument to the reviewers. This is what, this hasn't been done,
is really important to do. We know enough about it to know that the
research has been meaningfully bare on this problem. It's not just
totally nothing. We're building on something, so we can make a bet on
you in terms of your proposal. We want to make a good bet. A bet that
you're likely to find something that will be useful, okay. So, if you
can make that case, don't rule out any domain, just because somebody
else might be doing some work. You'd actually be surprised at how little
work is being done, if you compare it to the nature of the educational
problems that we have. It's really a drop in the bucket. So, I wouldn't
rule anything out on that basis.

Q: I've been trying to figure out how to word this without giving you a
full proposal. For many years, I've \[inaudible\] the National Science
Foundation. And, when doing research, one of the things that we got all
the time was a kind of conflict between evaluation and research. And,
every time you thought you were doing something good, it was evaluation.
So, what I'm really interested in is how much we have to attend to that.
For example, there's been a small number of very costly critical
thinking programs that are based on \[inaudible\] stages of development,
etc. And, the research is not clear on the effects of these programs on
the learning of mathematics, which is \[inaudible\]. If we wanted to
look at those programs and set up a situation where one could examine
what's going on in those critical thinking programs, vis a vis logic and
inference, etc., and the result the mathematics work, would that be
ruled out because it was deemed evaluation of a program that doesn't
exist anymore?

REYNA: I think it's a really good question. Again, I tend to be one of
these people who sees the links between things. So, as long as the
research really tested a hypothesis, as long a it really asked a
question---I mean, I think there are more or less sophisticated
questions. Questions that involve hypotheses about mechanisms are truly
more sophisticated than questions that don't, that just say, does this
work. So, you know, does this work is an outcome and it's an extremely
important and relevant outcome. That's the evaluation question. Does
this work or not. The mechanism question is how does it work, why does
it work, what, you know, how were these effects achieved. And, that's an
extremely important question, because then we know how to generalize the
work. So, I would say so long as you had, real research was going on, in
which real questions were asked that would advance fundamental
knowledge, adding this other component would seem to increase your
significance. So, I think it would not be something you'd rule out.

Q: Question about American Indian populations. Since there hasn't been
much research already on these populations, how will I be able to draw
on preliminary evidence if it's not there? Is there money in the grant
available to support initial research on these populations?

REYNA: Well, this is a big question. This is an area that's near and
dear to my heart, however. And, you and I should probably talk more in
detail. I can only give you a brief answer at the moment. In my
long-term view of these things, ultimately, this agency would have money
to do what's called high-risk, high-payoff research. NIH has such a
mechanism. High risk, high payoff means, you don't, maybe this isn't
going to pan out. But, it's so important. And, there's at least enough
information about feasibility. We can't just fund things randomly. That
isn't going to help anybody, right? So, you've got to have some basis
for believing that this is worth taxpayers' dollars. But, at the moment,
we do not have a mechanism of that sort. We have, in fact, you know,
okay, I will editorialize a little bit, we have remarkably little
discretionary research dollars in our budget. Remarkably little. We
don't compare to NIH on that score. And so on. Maybe some day that will
change. We're thinking of this, you know, this is the first step, and if
we're successful with the kinds of initiatives that we talked about here
today, if we can demonstrate that this will help, you know, the country
in the way that we hope it will, then we'll be able to use that as an
argument for more money. NIH has certainly been very successful making
very similar arguments. So, we'd like to do that.

At the moment, we do not have such a mechanism. Therefore, you are
leaving, I mean, you raised a lot of good questions implicitly even in
what you said. And, you and I should definitely talk. But, you leave
reviewers in a very difficult position. They have to sort of, they like
this, and they can say I'm going on faith, but, you know, you want
people to apply the same standards to the next application. What if they
say I like this other application? It just seems intrinsically good to
me even though there is no preliminary evidence, and I can't base it on
the usual standards. So, you have issues of fairness that arise then,
that are difficult. And, also, you know, you have to start somewhere on
research, I agree. The initial stages are always preliminary. But,
you've got to move past preliminary, and we've got to do it with great
respect and speed. And, I think the argument is when you have a real, a
problem of real life importance, of relevance, I would put this into
this category, this is an important issue, to look at the dropout rate,
and the academic outcome data, then you have to move to real science,
non-preliminary work, even more quickly. You have to have higher
standards. Because, it's so important, it's so urgent, you have to do a
really good job in this area, you have to do something that really will
help people. So, but, then, we have to deal with the realities that you
raised. And, I think that that really requires some more changes in this
agency and other agencies.

And, we're working on some things in the department. I'm on the
committee for that that's relevant to your concern, and one of the
reasons I'm on it is to try to do some of the things I just mentioned.
So, it's very important.