> **FY 2004 REPORT OF THE**
>
> **NSF COMMITTEES OF VISITORS (COV)**

**Date of COV: March 18-20, 2004**

**Cluster: Social and Political Sciences**

**Division: Social and Economic Sciences (SES)**

**Directorate: Social, Behavioral and Economic Sciences (SBE)**

**COV Overview: (prepared by Toby L. Parcel, Sociology, Purdue
University)**

The Social and Political Sciences Committee of Visitors (COV) met March
18-20, 2004 to review three NSF Programs: Law and Social Science,
Political Science and Sociology. Committee members divided along
disciplinary lines into three sub-committees to conduct detailed reviews
of the programs. The Committee benefited greatly from the assistance of
NSF staff including the several Program Directors and from the careful
and extensive documentation of program activities we were provided. We
appreciate this assistance without which the COV could not have
performed as efficiently as it did.

The Committee's charge was to assess the performance of the three
programs with regard to: (a) the integrity and efficiency of the
program's processes and management; and (b) results: the quality of the
outputs and outcomes of NSF investments. Our summary judgment based on a
careful review of the programs is that they are extremely well managed,
operate with a high degree of integrity, and are making important
contributions both to fundamental knowledge and to the betterment of
society.

Although the three programs are distinctive in important regards, they
share a number of important traits. While detailed reports are provided
for each individual program, it may be useful at the outset to identify
the most important similarities and to highlight common themes. We
itemize these below. Importantly, they are not organized by priority.

1.  There is consensus across the three program reviews that the basic
    proposal review process is fair and effective. Appropriate reviewers
    are chosen both to ad hoc proposals and to serve on panels; their
    advice is carefully considered and appropriately weighted in funding
    decisions. Both ad hoc reviewers and panel members are chosen from
    diverse groups of distinguished scientists. Proposal "dwell time" is
    reasonable, and has actually declined during the interval between
    the now and the 2000 COV. There is concern that there has been
    increased difficulty in actually obtaining an adequate number of
    reviews to evaluate proposals.

2.  Program Officers are dedicated, hard working, and skilled at
    managing the diverse tasks that are part of their job portfolios.
    There is concern, however, that Program Officers may be on the verge
    of "burnout" owing to high workloads, and that there may be
    insufficient resources to support their reaching out to relevant
    scientific communities to keep in touch with emerging trends in the
    fields. There is also concern that there needs to be an appropriate
    balance between permanent officers and rotators. Too frequent change
    of rotators could yield reduced efficiency owing to the need for
    frequent training, as well as loss of valuable experience needed to
    work within NSF. There is agreement, however, that rotators bring to
    the Foundation fresh perspectives vital to supporting new and
    important work in respective fields.

3.  The three programs have performed superbly in identifying, nurturing
    and funding science of the highest quality. NSF has supported a high
    percentage of the most innovative and highest impact work in these
    several fields. There is concern that funding shortages for the Core
    programs have often resulted in smaller awards of shorter duration
    than the merits of the proposals would suggest. Such a reality means
    that the funded projects may not be able to deliver the scientific
    returns that the quality of the science would permit if funding
    levels were higher. In addition, much meritorious research goes
    unfunded. These are very serious problems that, if not addressed,
    could have major negative consequences for project results well into
    the future.

4.  There is consensus that the dissertation grant programs are
    important aspects of the several programs. There is concern that
    managing what has sometimes become a large proposal flow has
    strained programs' resources. The COVs urge innovation in order to
    manage this aspect of the program efficiently and to encourage an
    appropriate balance among the many activities that Program Officers
    oversee.

5.  There is consensus that infrastructure investments have been an
    important part of innovation supported by the three programs. It is
    exciting to see the breadth of projects that reflect investment in
    infrastructure, and the excellence of scientific discovery that
    these investments encourage. The COVs note that investment in some
    infrastructure projects may preclude investments in others, and that
    the balance among these investments and those innovations stemming
    from non-infrastructure projects may need adjustment. Funding the
    largest infrastructure projects outside of the Program budgets
    appears to be the most feasible way to address this issue. This
    vital matter is also tied to concerns expressed in point 3 above
    regarding the sizes of awards.

COMMITTEE OF VISITORS ASSESSMENT OF THE NATIONAL SCIENCE FOUNDATION'S
LAW AND SOCIAL SCIENCES PROGRAM

Date of COV: March 18-20, 2003

Program/Cluster: Law and Social Sciences (LSS)

Division: Social and Economic Sciences (SES)

Directorate: Social, Behavioral, and Economic Sciences (SBE)

COV Members: Roy B. Flemming, Texas A&M University (Political Science)

> Carol Heimer, Northwestern University (Sociology) and American Bar
> Foundation
>
> Susan Hirsch, Wesleyan University (Anthropology)

Carol Swain, Vanderbilt University (Political Science, Law)

## INTRODUCTION

The problem of order has engaged many great minds, and legal systems are
one device that societies use to undermine a Hobbesian "war of all
against all" and to secure some kind of social contract among the
members of a society. Legal systems spell out rules about what people
(and other social actors) are required to do, what they are permitted to
do, and what they are prohibited from doing. They offer guidance about
people's responsibilities to one another, establish a framework for the
settlement of disputes, and lay down the ground rules for the allocation
of basic goods. Legal systems are singular achievements of human
society. They communicate the ideals of a society and offer instruction
in its most fundamental social and ethical norms.

Recognizing the importance of legal systems, the mission of law and
social science researchers is to ask searching questions about
variations in the legal arrangements of different societies, the
developmental course and trajectories of legal systems, the kinds of
divisions erected between formal law and other regulatory systems, the
ties between politics and law, the role of lay and professional decision
makers in formal legal proceedings, the processes by which legal systems
ossify, the legal mechanisms that secure and legitimate privilege, and
the variety of ways that social actors use their laws. The unique
contribution of this field is to reframe philosophical and
jurisprudential questions in the light of the concrete realities of
social life. Surprising things happen when the abstractions of legal
systems become tools in the hands of real people. Likewise, surprising
and important discoveries are made when scholars look for the
variability in legal tools, the uses that are made of those tools, and
the ingenuity and skill of users of legal tools.

Because legal systems represent our best collective attempt to bring the
needs and interests of the diverse members of a society into alignment,
the discoveries of sociolegal scholars are likely both to enrich our
scientific understanding of social life and to offer insights of great
practical importance. It is to law that we turn when we wish to regulate
interactions among individuals and institutions, control and respond to
violence, offer justice or redress to victims, or provide a check on
power and a vehicle for social change; likewise, it is to law that we
turn when we wish to symbolize or transform the social order. It is to
these key agenda that the research funded by the National Science
Foundation's Law and Social Science Program contributes.

Committee members were unanimous in their positive evaluation of the
Program. We paid careful attention to the criteria designated in the NSF
template for evaluating programs, though we also considered whether the
template was exhaustive. In assessing the Program, we examined a variety
of materials, including a random sample of LSS actions during the period
2001-03 (we looked at 59 jackets in total, including 30 awards and 29
declinations); all available folders on SGERs; several NCOVR reports;
the information compiled in loose-leaf binders, which included prior
project reports, Program statistics, and previous COV assessments;
supplementary statistical information supplied on the first meeting day;
and flip chart sheets of panel rankings of proposals. Some of these
materials were supplied by the Program Officer in advance or were
available when we arrived; other material was requested by committee
members as we deliberated. Although COV members were able to consider
the materials in private, the Program Officer made himself available to
answer questions and we spent a good deal of time in productive
discussion with him.

In general, we found the proposal review process to be efficient,
effective, and fair; the outcomes admirable; and the Program well-run.
Through the important research it funds the Law and Social Science
Program is making a key contribution to the development of knowledge,
while simultaneously making a broader impact on society. Below we
discuss our conclusions in detail and offer a number of suggestions for
improvements to the Program and its functioning. We are confident that
the Program will continue to facilitate excellent research, develop the
tools needed for further advancements in scholarship, foster and support
an increasingly diverse community of scholars in this still-growing
field, and have an impact on legal processes and society.

LSS Program research contributes to the growth of scientific knowledge
in such areas as the development of international legal institutions,
the spread of the rule of law, the control of crime and violence whether
it be domestic or international, the regulation of key industries, the
work of judges and juries and police in processing criminals, and the
use of legal tools to control powerful political elites. Moreover, at
this historical juncture, such research is of enormous practical
importance as legal systems offer irreplaceable tools for helping to
shape and regulate the often disorderly transition to a more global
world.

**PART A: INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

*A.1 Quality and Effectiveness of Program's Merit Review Procedures*

[Overview]{.underline}. We assessed the appropriateness, efficiency, and
effectiveness of the review mechanisms through examinations of the
proposal jackets, discussions with the Program Officer, and evaluations
of the reports on completed projects. Individual COV members scrutinized
the jackets before discussing them as a group.

The COV was impressed by the quality of the LSS Program's review
process. In the vast majority of the cases, the ad hoc reviews, reviews
by panelists, panel summaries, and Form 7s addressed core issues
concerning the adequacy of the research design, the relation of the
proposed research to previous scholarship, the qualifications of the
investigator and co-PIs, and the importance of the likely intellectual
contributions and broader impacts of the proposed work. The COV was
equally impressed by the fact that Fastlane allows PIs whose proposals
have been declined to view the ad hoc reviews and assessments by the
panel reviewers in a timely fashion; this is a very welcome improvement
in the transparency and efficiency of the review process.

Briefly, the COV felt that the review mechanism is appropriate, the
review process efficient and effective, and that reviews are consistent
with the Program's announced criteria and priorities. The reviews were
quite thorough, and every jacket we examined included at least a couple
of very detailed reviews that provided substantive feedback to PIs and
guidance to the panel and Program Officer. Panel summaries are generally
of high quality, although some convey less information or are less clear
than others. This may be an artifact of the use of the "scribe" method
for some review rounds. The current Program Officer has taken on the
task of preparing panel summaries, which should alleviate these
concerns. The last COV recommended that panelists who also review a
proposal comment directly on the ad hoc reviews that contradict the
panelist's own impressions and that comments that explain the divergence
in reviews be included in the panel summary. In many cases panel
summaries focused on important issues in a way that would help PIs
understand the panel recommendation and ultimate decision about funding.
Time to decision (dwell time) has been going down and in 2003 dropped to
just over 4 months, a very reasonable number.

The COV concludes that the LSS Program's proposal review process is
functioning very well with only minor adjustments needed. Below we raise
some concerns about review criteria and transparency and tracking of
final funding decisions (in the post-panel meeting period).

[Reliance on Fastlane]{.underline}:

Fastlane appears to have improved the efficiency of the review process,
at least insofar as dwell time is concerned. The average dwell time for
proposals decreased from 5.56 to 4.02 months between 2001 and 2003, and
the proportion of proposals getting through the process in six months or
less increased from 81% to 99% during that same period.

We do worry, though, about those scholars who may find the increased
technical requirements for grant submission onerous. Although we have no
reason to believe that Fastlane presents a special barrier to any
particular group, we worried about discouraging submissions from
potential submitters in institutions without strong technical support or

where the scholar works in a community without colleagues who have
previously submitted or reviewed proposals. Although we usually worry
about younger scholars and how to build capacity, we do not want to
ignore the importance of keeping experienced older scholars, who may on
average be less computer savvy than younger scholars, involved in
proposal submission.

[Declination Decisions]{.underline}:

NSF clearly should decline unworthy proposals. However, if declination
rates are too high with the result that strong proposals go unfunded,
the efforts of PIs, ad hoc reviewers, and review panelists will have
been wasted. Likewise, declining strong proposals that then reappear in
the queues of subsequent years means that we are multiplying the time
spent on submission and review at the expense of other activities,
perhaps with little gain in quality in the intervening period.

The majority of proposals do not receive funding, and that is probably
as it should be because the review panel judges these proposals as "not
fundable." From our discussions with the Program Officer, it seemed that
all of the most worthy projects are funded (i.e., proposals that he and
the panel ranked as "must fund" or "should fund"), but that only
approximately a quarter of "could fund" proposals receive funding. It is
in this latter category of "could fund" applications that the Program
Officer must exercise discretion, balancing disciplinary representation,
the riskiness of projects, budget constraints, and the like. The Program
Officer might consider discussing with the panel plans for exercising
that discretion and thereby soliciting panel members' views as to
whether any particular priorities are warranted in that proposal round.

[Panel Funding Priorities]{.underline}:

The COV thinks that more systematic and complete data on the funding
priorities established during the review process is required. We would
expect some variability in the composition of the pool from one cycle to
another, and indeed the panel discussion sheets (from the flipchart used
by the group) indicate that that is the case. The most recent cycle
apparently contained an unusually large number of high quality proposals
with slightly over half of them being deemed "could fund" or better by
the group. If only a quarter or less of "could fund" proposals receive
funding, then we might expect to see a high proportion of these research
projects reappearing in the queue in subsequent years. Counts of the
proposals placed in this and other categories by the panels in the
course of their deliberations would have facilitated our assessment of
the process and made it easier for the COV to form a judgment regarding
the adequacy of the Program's overall funding.

[Project Duration and Budget Constraints]{.underline}:

Project duration, like declination and resubmission rates, gives some
information about the balance of time spent on proposal preparation and
submission as compared with time conducting research. It also gives an
indication of whether the length of time projects are expected to take
might be adjusted to match funding opportunities. Recognizing that there
is some diversity across fields in what constitutes a "project" (e.g., a
project that would result in a book or collection of articles vs. one
that would result in a single article), how long projects endure, and
how many projects a PI is involved in simultaneously, we applaud the
recent move in the direction of increased duration. In 2001 and 2002,
the respective mean duration of awards was 1.86 and 1.69 years; the mean
duration for 2003 by comparison was 2.21 years. The COV encourages the
Program Officer to see, if it is possible, whether budget constraints or
limitations may lead to "under-funding" of proposals by shortening their
duration and perhaps thereby inappropriately diminishing the scope or
scale of projects.

*A.2 Implementation of NSF Merit Review Criteria*

[Overview]{.underline}. In general, individual reviews (both ad hoc and
panelist) and panel summaries addressed both merit criteria. The
previous COV noted that individual reviewers routinely failed to mention
the second criterion (broader impact) but anticipated that reviewers
would become accustomed to what was then a relatively new question in
the review process. Apparently, they have adjusted. The review analyses
(F7s) also appropriately note the contributions of proposals to both
merit criteria.

However, the comments of ad hoc reviewers as well as those by panelists
suggest that reviewers treat the two merit criteria differently with
more attention directed toward intellectual merit and, most likely, more
weight on that criterion in the rating and decision. For weak proposals
the lack of intellectual merit generally leads to a briefer discussion
of broader impact or to the conclusion that no impact would result from
the project as currently proposed, given its intellectual limitations.

[Assessing Impact]{.underline}:

We applaud the increased attention to broader impacts of research, but
we wish to highlight a minor concern about this criterion. Although
scholars should be attentive to the broader impacts of their work, this
is not, in general, their forte. We should remember that "impact"
requires not just the attention and effort of the researcher but the
receptiveness of those who might be affected, which suggests that our
ambitions in this respect should be realistic and tempered by modesty.
In contrast to the modesty we advocate in assessments of broader
impacts, however, we urge reviewers (and indeed proposers) to be bold in
their thinking about the intellectual impacts of proposals and
NSF-funded work.

Keeping in mind that NSF\'s core mission is basic research, reviewers
and PIs should think about social processes broadly and look for
connections between work in one substantive area and comparable work in
other substantive areas that will foster cumulative, scientific
progress. When conceived properly, generality and broader impacts are
not inconsistent ambitions. Research on terrorism, for instance, may
seem especially likely to have an immediate impact, yet if formulated in
generalizable terms could tell us a great deal about the phenomenon of
extreme events that are not stochastically predictable.

*A.3 Selection of Reviewers*

[Overview]{.underline}. The COV considered several issues concerning the
selection of reviewers. We believed that on average the proposals were
sent initially to an adequate number of reviewers and that reviewers
possessed the appropriate expertise and qualifications. Overall,
reviewers are affiliated with a wide range of institutions, with
variability in geographic location, size, and type (e.g., major research
institution, smaller teaching-oriented institution). The top ten
institutions that provide reviewers are major research universities. The
inclusion of at least one reviewer from outside the PI's primary
discipline --- as is the case for many of the proposals --- is
appropriate given the interdisciplinary nature of law and social science
research.

It is difficult to determine whether COI procedures were followed in
every case in the selection of reviewers and during the review process.
Correspondence in the jackets indicates that reviewers routinely recuse
themselves for COI-related reasons; F7s indicated that panelists with
COIs left the room appropriately during deliberations.

Although we were on the whole satisfied with the selection of reviewers,
we are mindful of the effort required to solicit and process reviews.
For this reason, we urge that the Program Officer consider modifying
current procedures so that the number of reviewers (beyond the required
three) varied with the requested amount of funding. Below we offer
several suggestions with respect to this issue and, as well, advocate
increasing the diversity of reviewers.

[Minimum Number of Reviewers]{.underline}:

We understand that --- with the exception of SGERs --- proposals must
have at least three reviewers (typically with two of these being panel
members). Although we remain concerned that an adequate number of
reviews always be obtained, we would urge some movement in the direction
of varying the number of reviews with the size of the requested award.
Given the dollar limit on dissertation proposals, however, we believe it
might be wise to require and thus request fewer ad hoc reviews for these
submissions. Proposals requesting larger sums merit extra reviewer
attention, we would argue, on the grounds that they pose special
opportunity costs in other research foregone. Ideally, proposals
requesting the largest award amounts should have the highest number of
outside reviewers, i.e., three or four. We make specific recommendations
for reviewing dissertation research below.

[Return Rate of Ad Hoc Reviews]{.underline}:

Although proposals are routinely sent to an adequate number of
reviewers, the number of reviews returned to the Program is often low.
We are especially concerned when proposals received only three ratings,
two of those by panel members. Setting aside for the moment the question
of dissertation proposals, we recommend that each proposal have ratings
and/or comments from at least two outside reviewers at the time of panel
evaluation. In at least one case, a proposal was declined on the basis
of one outside review and two panel reviews. It was puzzling that none
of the six reviewers suggested by the PI --- all with expertise in the
field --- had been approached for reviews. People with a substantive
familiarity in the field of the project should review proposals.

We recognize that identifying reviewers, maintaining a list of
reviewers, and securing reviews from them are burdensome for the Program
Officer. COV members noted this burden, and the frequent failure of
reviewers to respond to requests for reviews. Certainly, reviewers
suggested by PIs should be approached when appropriate and panel
members, former panel members, and central figures in the field be
contacted for additional suggestions of appropriate reviewers. We note
that the previous COV recommended additional support staff to assist in
data collection and entry with respect to this and other tasks of the
Program Officer. We also endorse this recommendation.

# Reviews of Dissertation Proposals:

The COV recommends major changes in how the Program processes
dissertation research. Although external reviews of dissertation
proposals may secure valuable advice and some early augmentation of
professional networks, it is not clear that the LSS Program can afford
the luxury of multiple reviews of small dollar proposals given its
(appropriate) practice of resting reviewers for a cycle after one in
which they submit a review. To address the problem, the COV recommends
that dissertation proposals be treated differently with respect to ad
hoc reviewers. If the number of proposals sent to reviewers is decreased
overall, then more reviewers would be available and would not themselves
be so burdened with requests. It may be that decreasing the number of
proposals to be reviewed by ad hoc reviewers would help alleviate this
burden.

We imagine two possible scenarios: 1) Each dissertation proposal would
be reviewed by three panelists. This strategy would have the advantage
of completely eliminating the burden of dissertation reviewing on ad hoc
reviewers, who could then be used for review of regular proposals. The
corresponding disadvantage would be the extra work for panelists and the
lack of feedback from ad hoc reviewers. 2) Each dissertation proposal
would be reviewed by one outside reviewer and two panel members. The
advantage of this strategy would be decreased use of outside reviewers
with no increase in panelists' work. The disadvantage would be slightly
less feedback to the panel and to graduate student PIs.

[Gender and Ad Hoc Reviewers]{.underline}:

The COV noted that, in the pool of proposal reviewers, female reviewers
constituted less than half of the external reviewers (855 of 2800
reviewers with the gender of 88 unknown). Although the number of women
increased each year, additional improvement is warranted, and the
Program Officer should take steps to augment the number of women from
whom reviews are requested.

[Race and Ethnicity and Ad Hoc Reviewers:]{.underline}

There are no statistics available for the number of reviewers from
underrepresented groups. We note, however, their underrepresentation
among PIs. This should be addressed by, among other strategies,
utilizing the minority databases of major professional organizations to
recruit additional reviewers.

*A.4 Portfolio Assessment: Managing Diversity*

[Overview]{.underline}. The COV found the quality of the projects to be
impressively high, encompassing a wide range of domestic and
international topics. Diversity is evident in the types of projects
funded both at the dissertation and the standard grant levels.

The program supports basic, cutting-edge, and multidisciplinary research
focused on major theoretical questions, the answers to which generally
have broad importance and impact. The quality of this research means
that project findings are published in premier, peer-reviewed
professional journals and by major university presses. Dissertation
grant proposals constitute a significant portion of the projects seeking
support; roughly 20% of all submissions are requests for dissertation
support. Dissertation awards constitute key investments in the Program's
research communities, foster professional training in the communities'
research methodologies, and support the continued vitality of these
communities. It should be noted in this context that Research
Experiences for Graduate Students, which are supplemental awards to
standard grants, offer a means of supporting and mentoring the training
of students in the study of law and law-like institutions. The COV,
however, believes that the Program should encourage the submission of
more RUIs and REUs to bolster its educational mission.

### A4.1 --- An Assessment of the General Portfolio

[Size and Duration of Awards]{.underline}. The dollar amounts of NSF
awards have steadily increased over the past three fiscal years from a
mean of \$98,390 in 2001 to \$111, 499 in 2003. Although the scope of
the projects is difficult to measure with precision and thus difficult
to relate to the size and duration of the grants, one-year grants
account for roughly one-third of all standard grant awards. This raises
concerns that PIs may trim the scope of their proposals to fit
expectations of funding opportunities shaped by the Program's efforts to
fund as many equally competitive proposals as possible. We note as well
the apparent reluctance of the Program to fund the few proposals that
request exceptionally large sums. We comment below on the opportunity
costs of funding extremely large projects, proposing increased scrutiny
(perhaps focused directly on opportunity costs) with increases in the
size of the requested budget.

[Risk and the Portfolio]{.underline}. One example of high-risk proposals
is Small Grant Exploratory Research projects. Over the past three years,
the Program has funded nine of these proposals with award amounts
varying from \$3,700 to \$75,000. While some of these proposals dealt
with the 9/11 episode, others take creative advantage of events to
explore important socio-legal questions. For example, one SGER, prompted
by a recent lethal fire in a Moscow theatre, quickly put into a place a
survey to inquire into the dynamics of litigiousness among the
survivors, who under the former Soviet regime would not have had
recourse to the courts and law to challenge the government for its
negligence. The SGER proposals provide the Program with a mechanism to
respond to fast-moving, often unique and thus unexpected events that
create rare opportunities to investigate socio-legal phenomena for which
evidence quickly decays over time.

From a broader perspective, the Program funds basic research which
invariably has varying degrees of risk, for instance, from
theoretically-derived questions that may be left unanswered or partially
answered at the conclusion of the research. Research projects that
employ methodologies adopted from their respective communities' "normal
science" approach that seek to confirm and extend current knowledge are
a valuable part of the Program's portfolio. At the same time, we believe
that the Program should be encouraging innovative designs and projects
that challenge conventional wisdom. At the present, we believe that
although the normal review process does not preclude funding of these
projects, it also makes no special effort to single them out. As an
example of the kind of project we have in mind, we point to a funded
dissertation project that for the first time looks systematically at the
"unpaid" petitions for judicial review in the U.S. Supreme Court. The
widespread and accepted view is that these petitions, which account for
over three-quarters of the requests for review received by the Court,
are "frivolous," lack legal merit, and are rarely granted. The project's
preliminary findings seriously challenge the conventional wisdom,
although it could have just easily been confirmed.

[Prior Experience of PIs]{.underline}. The total number of standard
grant submissions by new investigators (222) over the past three years
is roughly 50% higher than the number of submissions by prior PIs (144).
The award rate, however, is 15% for new investigators versus 40% for
prior PIs. While the funding rate for new PIs has improved marginally
from 13% to 18%, the rate for prior PIs has fallen from 51% to 31%.
These changes suggest that the advantage of "repeat players" has
declined. However, the data are ambiguous because the "prior PI"
category includes both successful and unsuccessful applicants; the
inclusion of the latter applicants may mask a stronger relationship
between prior success and current success. With this caveat, it appears
that the portfolio is appropriately balanced with regard to awards to
new investigators.

[Regional, Institutional, and Disciplinary Balance]{.underline}.
Although the percentage of funds going to ESPCor states declined from 23
percent in 2001 to 15 percent in 2003, the COV found no regional bias in
the awarding of grants. The distribution of grants seems appropriate
given the number of submitted proposals per region. For example the
higher success rates for the Farwest (30.56 percent), Great Lakes (34.33
percent), and the Mideast (25.32 percent) correspond with larger numbers
of proposal submission coming from those regions. In fact, over half of
the 337 proposals (218) submitted came from one of those regions.

The COV found that major research universities generate a high
percentage of the submissions. We observed a rough correspondence
between the frequency of submissions and the success rate for funded
proposals. By contrast, declinations are common for PIs, even previously
successful PIs, from major research institutions. This pattern, given
the resources and quality of the institutions, is not surprising. The
portfolio balance, thus, appears justifiable.

The Program funds projects representing a broad range of disciplines and
subdisciplines. Funded projects have included research on international
crimes and tribunals, naturalization of refugees, women's suffrage as a
social movement, homicide rates in Mexico and the United States, as well
as constitutional consciousness in Russia.

[Minority Participation]{.underline}:

While the balance of the Program's awards across institutions,
disciplines and region appear satisfactory, the COV is concerned about
the continuing under-representation of minority submissions and awards.
Of the 373 proposals submitted between 2001 and 2003 only 20 came from
minority PIs. Although we recognize the new constraints posed by changes
in the legal and political environment surrounding affirmative action
initiatives, we nevertheless believe that the Program can and should do
more to improve these numbers. The COV urges continued vigorous outreach
efforts by the Program to these communities of scholars. Aggressive
Program outreach could take the form of special workshops designed to
smooth the transition from the minority training programs that NSF has
sponsored in the past to more active participation of racial and ethnic
minorities in the core process of proposal submission and review (and of
course carrying out funded research).

The COV was favorably impressed by the participation and success of
female PIs. Between 2001 and 2003, 31.45 percent of the proposals
submitted by women were funded while men had a success rate of 24
percent. Nevertheless, the mean awards for females in each of the years
examined was lower than the mean award granted males.

The Program is responsible for meeting the research interests and
supporting the methodologies of diverse academic or disciplinary
communities. The management of these responsibilities requires
sensitivity to and understanding of the research traditions and
emergence of new ideas within these communities. Available data
unfortunately make it hard, though perhaps not impossible, to assess
whether the Program's portfolio at any one time, or over time, strikes
the right balance.

[Small Grant Exploratory Awards (SGER)]{.underline}:

The COV, while impressed by the flexibility and quality of SGER
proposals, is concerned that, despite their currently small number, they
are not peer-reviewed. The COV recommends that whenever time permits,
the Program Officer should solicit ad hoc or panelist reviews of the
proposals. It also recommends that any proposals that are funded without
peer review be discussed at the next session of the Program's review
panel.

[Diversity of Methods]{.underline}:

Projects funded by the Program make use of a wide variety of research
methods ranging from experimental studies, statistical analyses of
survey data, interviews conducted on small and large scales, examination
of archival materials, and ethnographic fieldwork, among others. More
than a few projects rely on multiple types of data collection and
analysis. While noting that, in 2001, the Program funded
non-dissertation projects that were primarily quantitative in nature at
about the same rate as those that were primarily qualitative in their
research methods, the COV identified a trend toward funding quantitative
projects at a higher rate in two subsequent years. We are not overly
concerned about the ratio in those years and are unable to determine
whether such a disparity will be significant in the future. Quantitative
and qualitative dissertation projects were funded in roughly equal
numbers in each of the three years. Given the Program's long history of
supporting qualitative research, we would urge that the Program Officer
remain attentive to the needs of this constituency of the research
community and continue to encourage such submissions.

*A4.2 --- National Consortium for Violence Research and the Program's
Portfolio*

Over the last few years, NSF has had to grapple with the question of how
the National Consortium for Violence Research (NCOVR) should fit into
the Program's portfolio. The Consortium has had considerable success in
capacity building, and we applaud this success and welcome its attempts
to foster a multi-disciplinary, multi-level focus. Despite the promise
of the enterprise, NCOVR has posed some complex managerial problems with
which the Program and directorate have had to grapple. The COV concurred
that these problems have been handled in a reasonable fashion, and we
were pleased that a compromise has been worked out that will allow NCOVR
to continue with the most successful of its activities (i.e., capacity
building) for a limited period.

The COV concludes that it is important to build on NCOVR's initial
efforts and to redirect the energies of this now-expanded community and
its pool of resources to basic scientific research on violence.
Fundamental questions remain about how violence arises and is supported
or controlled by persons, situations, communities, and institutions. We
would argue that, in this field especially, important policy questions
need to be put in the context of basic research, and that the LSS
Program is the appropriate home for this research initiative. Indeed,
there seems to be no well-defined division between research that falls
under NCOVR's mandate and the kinds of proposals normally submitted for
funding by the LSS Program. As we have examined the review materials, we
have come across a number of research proposals and research reports in
the LSS lists that could just as well have been in the NCOVR files:

-   Fagan's research on "Situational Contexts of Gun Use by Young Males"
    (9515327)

-   Lockwood's work on "Violent Incidents Among African-American Public
    School Students: A Proposal for Research" (9727882)

-   Loftin's "Establishing an Agenda for Science and Research on Serious
    Assaultive Violence" (9415124)

-   Monkkonen's "The Age Structure of Homicide: Stable or Variable"
    (9809846)

-   Roth's "Why Northern New Englanders Seldom Commit Murder: Crime and
    Violent Death in New Hampshire and Vermont" (9809050)

And among funded regular research proposals:

-   Matsueda's "Racial Heterogeneity, Neighborhood Controls and
    Violence" (0004324)

-   Monkkonen's "Age Standardized American Homicide Rates" (0111725),

-   Philips's "Understanding Variation in U.S. Homicide Rates across
    Time and Space, 1970-1996" (0111542)

-   Schatz's "Undoing a Violent Past: Explaining Electoral Homicide in
    Mexico's Democratization" (#0004402)

And among funded dissertation proposals:

-   Mills's "Counseling and Advocacy Services for Intimate Abuse
    Victims: A Study of Recidivism in a Mandatory Prosecution
    Jurisdiction" (00043400)

-   Cancian's "Domestic Violence and the Family Man: Courtroom and
    Community Narrative About Domestically Violent Fathers in Rural,
    Urban and Suburban Settings" (02111778)

-   Jordan's "A Multi-Systems Violence Prevention Program for Girls"
    (0137096)

This list of proposals suggests to the COV that there is merit in making
more permeable the disciplinary and institutional boundaries that divide
violence research from other LSS research, both with the aim of looking
closely at the overall pool of proposals for research on violence and
enlarging the mission of the violence research community. In particular,
we would advocate a deeper consideration of the ties between individual
and collective violence so that we begin to understand more fully the
complete package of violent behavior whether it be at the level of
violence between intimates, violence that occurs as part of criminal
activity, or violence with political purposes.

During this transition period, as the LSS Program reconsiders the
successes and challenges associated with NCOVR, we urge a collective and
relatively public assessment of achievements and a consideration of how
these achievements create a foundation for future research. Such a
reassessment might profitably be organized, for instance, as an advisory
conference that would consider both a research agenda and how best to
fit the activities and projects of the "violence research community"
into the LSS portfolio. The COV believed that some earmarking of funds
for violence research or an ongoing series of conferences might be
appropriate, particularly to set a new agenda for violence research. We
would not be supportive of the establishment of a new center to replace
NCOVR.

*A.5 Program Management*

[Overview]{.underline}. The COV concludes after its review that the
Program is effectively and efficiently managed given its resource
constraints. As these constraints are likely to grow increasingly
confining in the future, consideration should be given to the changes
that are outlined below and elsewhere in this report.

The COV also feels that the Program is flexibly managed to respond to
changes in the interests of its research communities. In most ways, in
fact, the Program's review process encourages responsiveness and
attention to these interests. As a consequence, the Program's portfolio
in its most important respects reflects the priorities of its
constituent communities.

The COV nevertheless has several concerns about the Program's
management. They are listed below.

[Administering the Review Process]{.underline}:

As noted earlier in this report, the COV worries that the present
practice of sending all proposals, with the exception of SGER proposals,
to ad hoc reviewers is a time-consuming and perhaps unnecessary use of
administrative resources. One of the chief challenges facing Program
Officers, which is compounded in part by the rotating nature of their
position and the multidisciplinary nature of the Program, is finding
skilled and substantively appropriate ad hoc reviewers. As about 20
percent of submissions are for dissertation proposals, the COV strongly
suggests that consideration be given to revising the review process of
these proposals so that it relies more heavily on panel reviewers.

The COV feels the process can be revised without jeopardizing the
integrity of dissertation proposal reviews. Moreover, we believe that
such a change have the additional salutary effect of increasing the
Program Officer's capacity to do needed outreach (especially with
minority communities) by freeing up more of his time. We note that other
programs have made comparable changes in the review process for
dissertations proposals. To the extent that additional funding for staff
would assist the Program Officer accomplish these goals, we endorse an
attempt to allocate such funds.

# Facilitating the Committee of Visitors (COV) Review:

COVs are convened every three years and are asked to assess the
performance of the Program according to the rubrics and measures as
outlined by the current "NSF FY 2004 Core Questions for COVs." Despite
the helpfulness of the Program Officer, at several points during its
deliberations, the COV found its efforts frustrated by the absence of
relevant information, disorganization of information, and more simply
the lack of a spreadsheet (organized by proposal) with the appropriate
data. While review of the jackets and some of the summary statistics
used by the COV were helpful and would continue to be helpful, the COV
feels a spreadsheet with decisions, rankings, number of reviewers,
received reviews, discipline, institution, etc., would facilitate the
search for information needed to address the "Core Questions" and
necessary for the timely completion of the COV's report.

[Program Officer Position]{.underline}:

The COV also discussed whether the current policy of a "rotating"
Program Officer drawn from one of the Program's constituent research
communities is appropriate. It also considered the wisdom of the recent
"trend," if that is what it is, in which four of the last five Program
Officers have been drawn from a single discipline (e.g., political
science). The COV wishes to be clear that its concern with disciplinary
diversity in this central leadership position should not be construed as
questioning the credentials, skill, energy, or fairness of the
individuals who have held this position. Nor is the meritocratic process
used by the Foundation in doubt. We were favorably impressed by the
current Program Officer. Nonetheless, the diverse character of the
Program's constituency suggests that every effort should be made to
solicit applications from candidates from a variety of disciplines and
--- insofar as this is possible given the pool of qualified applicants
--- to select an individual from some other discipline when the position
of Program Officer next becomes vacant.

As for the policy of a "rotating" Program Officer, the COV feels, on
balance, that the regular "renewal" of leadership brings fresh
perspectives and new energy to the Program. These merits outweigh the
start-up costs of learning the job, developing an understanding of the
LSS research program, and forming crucial ties with the entire
constituency.

**PART B. RESULTS: OUTPUTS AND OUTCOMES OF NSF INVESTMENTS**

*B.1 --- Outcome Goal for People*

The LSS Program continues to make significant investments in basic
research that dramatically expand human capital in the American social
sciences. A previous COV noted that the Law and Social Science Program
provides a "disciplinary home" for scholars who study the legal system.
Fostering the research of that group of scholars, and adding to the
group, are Program goals that should be encouraged and supported. The
Program's role in funding dissertation research is critical to the
development of capacity in the field, particularly its support for
interdisciplinary projects that might be deemed ineligible for
disciplinary-based funding. More needs to be done to foster diversity in
the law and social science community, including by supporting research
by scholars from underrepresented groups.

It is especially noteworthy that the Program has extended its reach
cross-nationally, connecting scholars from outside the U.S. to the
research and professional activities of those in the U.S. The COV sees
great benefit in the elaboration and development of comparative
perspectives of law and society that bring together and create a
transnational community of scholars. Kidder's (0137076) "Symposium on
Advances in Law and Civil Society in the Pacific Rim" and Munger's
(9911507) "Workshop on International Research Collaboration in Law and
Social Science" are examples of how the LSS Program can support the
emergence of a transnational community of scholars commensurate with the
current globalization of law and law-like regimes.

*B.2 --- Outcome Goal for Ideas*

Understanding legal systems (including their diverse forms and their
articulations with other institutions) and assessing how they function
and how to sustain and improve them are central endeavors of research
funded through the Law and Social Science Program. Researchers analyze
how courts function, scrutinize the content and implications of legal
decisions, and examine law's role in shaping and reflecting social life.
As one component of their explanations of how legal systems operate,
scholars also study the lives and work of legal professionals (Sarat
9818366, Garth 0115521).

Across every cultural context law and law-like institutions provide a
context and set of mechanisms for debating and resolving broad questions
of justice. Research funded through the Law and Social Science Program
examines law's role in the containment, management, and resolution of
disputes at all levels of society. The Program has supported an array of
projects related to domestic and international crime and violence. These
studies have included surveys related to the September 2001 terrorist
attacks (Skita 0210053, Silver and Davis 0140541, and Moore 0201149);
research related to race and community policing (Matsueda 0004324,
Edelman 0321725, Seron 0096476); and research on homicides rates and
patterns (Monkkonen 0111725, Schatz 0004402, Philips 0111542, Snajdr
0004402). As baby boomers' offspring enter the age groups most likely to
commit violent crimes, studies of crime and violence are likely to be
especially important.

Law and society researchers have not shied away from those highly
controversial topics that reveal severe cleavages in normative and
ethical perspectives, such as the American death penalty, racial
discrimination in legal institutions and in society more generally, the
treatment of juvenile offenders, the fate of civil liberties during the
war on terror, the role of the judicial branch in determining morality
or effecting social change, whether institutions can be trusted to
self-regulate.

Research on the relation between law and culture remains an important
endeavor for law and social science scholars. Legal culture, legal
concepts, and legal procedures pervade our institutions and
organizations and offer the rationale for their operation. Law's
relation to norms and ethics is an important area for the empirical
study of culture and law. Research on the legal consciousness of
ordinary people --- including what they think about law and what they
expect from their legal system --- are also relevant concerns taken up
by scholars of law and social science. Such questions of legal
consciousness are crucial to determining whether people believe law is
accessible to them, how they use law, and whether they believe they
should accept legal decisions even when they are disadvantaged by those
decisions. Heuer (9710946) joins a long and productive line of research
on procedural justice.

Behind much law and social science research on the procedures for
redressing grievances lies the question of whether legal processes could
be made more efficient, more rational, less opaque, and more congruent
with public expectations and shared norms. Here law and social science
research has as much to contribute to those considering reforms in
policy and practice as to those pursuing more scholarly questions.

These broad questions about justice often span national borders. In
recent years the Law and Social Science Program has made a conscious
effort to encourage the study of transnational legal processes. As the
globalization of the economy, technology, and cultural expression
continues, law will change in tandem. We know that these processes have
significantly altered the form and function of nation-states, calling
into question such well-established law-related concepts as sovereignty,
citizenship, and jurisdiction. Whether these processes result in a
coalition of nations, a set of regional powers, or an empire, we will
need to understand law's changing role. Attention to the new ways in
which the U.S. legal system articulates with other legal institutions
and processes is already established and significant attention is being
devoted to new regional and international legal bodies.

Law, and the much debated concept of the "rule of law," are central to
any functioning democracy. Law articulates the responsibilities of those
living in a society and designates their relationships to one another
and to the government. It delineates the statuses individuals occupy and
offers the means of contesting identity and social roles. Thorny debates
over, for example, who is a citizen, who is a juvenile, or who is a
refugee, and what rights and obligations these groups have, are sites of
productive law and social science inquiry. In establishing rights and
responsibilities, law is also the site for appeal as people seek to
extend the benefits of law beyond elites. Law's role in the empowerment
of marginalized groups is a concrete means through which citizens can
hold a government accountable through an avenue separate from political
processes. Law and social science research has long explored law's
potential and limits as a tool for reshaping society. This tradition has
been reinvigorated by exciting new research examining the role of law in
newly emerging democracies. In periods when the legitimacy of legal and
political institutions comes under attack, research on the rule of law
is especially important.

The past several years have seen the growth of NSF-funded research on
constitutions, constitution-making, citizenship, and rights. Much of
this work focuses on the international arena. Arjomand (9601357)
examined constitutional law and the modernization of political
institutions in the Islamic Middle East and Scheppele\'s (0111963) study
focused on rights in relation to the development of Constitutional

consciousness in Russia. Scheppele also received a SGER award to study
the creation of a new Hungarian Constitution (9514174). McCann\'s
(0318497) recent project examines legal pluralism in relation to
individual rights and Sassen (9906170) studies the human rights
movement. Several dissertation projects also contribute to this area by
examining the Human Rights Commission in Sri Lanka (Darian-Smith
0136676) and citizenship rights and the politics of race in Britain
(Rose 0211195), human rights and international legal practice in Africa
(Barnes 0318305), participatory constitution-making in Africa (Widner
0001748), and legal rights in relation to ethnic/racial identity in
Brazil. This general area of inquiry is markedly transnational in scope.
In many instances the questions explored are the subject of active
debate in the societies and institutions studied. By tracking world
events as they are happening, this research has the potential for
significant impact on the evolving discourse of rights and the
development of constitutions in a diverse group of societies.

As new transnational forms of governance emerge, sociolegal scholars
have also been mindful of the need to consider the boundary between
formal law, other systems of rules, and other regulatory cultures. With
support from the Program, scholars are moving the study of regulation in
new directions. We note, for instance, that Silbey's (0216815) project,
"Safe Science: Governing Green Laboratories" examines the regulation of
laboratory science at major research universities in a sometimes
cooperative and sometimes conflictual joint venture between government
regulatory bodies such as the EPA, university administrators and risk
managers, and professional associations. Other related work on emerging
forms of regulation also considers the legal cultures of institutions
(Fuller 9511843, Edelman 9696847) and the extension of the joint
legal/professional regulatory regimes across national borders (Heimer
0319560). In a similar vein, Rikoon Sanford (0004300) supervised a
dissertation that promises a broad interface between science and law by
examining the role of science in environmental decision-making.
Reviewers were optimistic that Sanford's supervised study would provide
practical insights for both physical and social scientists.

The LSS Program\'s significant support to research in the psychology of
law continued in 2001-2003 with the funding of 16 projects ranging from
jury studies, to studies of identification and memory in the justice
system, and research on technology\'s influence on decision-makers in
legal contexts. Completed projects include research on, among other
topics, the admissibility of scientific expert testimony (Penrod
9618580) and the evaluation of hearsay evidence (Thompson 9409681).
Several studies add to the distinguished record of research on
understanding juries, including one on jury nullification and one on
juror reasoning. These large scale experimental studies have shed light
on the process of jury deliberation, including how individual jurors
understand evidence and law and also how the jury operates as a
deliberative group. A project by Penrod (0137725) takes jury research
into the new topic area of risk management. Studies of jury
decision-making in capital cases draw on insights from jury research to
address a critically important issue of our time (Penrod 9810039, Weiner
0196301). Diamond\'s study (9818806) of actual civil juries that were,
for the first time, video-taped during deliberations, inaugurated a new
direction for research, one which has gained media attention and has
already influenced public discourse about juries and shaped policy
considerations about the role of media and technology in legal
processes.

Projects examining aspects of perception and cognition (e.g., eyewitness
identification, facial recognition, perceptual memory, developmental
differences in perceptual and cognitive function) also continue
well-established lines of research in psychology and law. A series of
studies have compared the ways in which police conduct suspect line-ups.
NSF-funded studies have determined that the traditional line-up, which
presents a

group of suspects to the witness, more often results in errors of
witness perception and memory than is the case with line-ups conducted
by presenting suspects sequentially. These findings have captured the
attention of the media and policy-makers and led to changes in police
practices in some jurisdictions. Recently funded projects (Gronlund
0204182, Ross RUI0321592) will extend this research area. The outcomes
in the area of law and psychology are positive and have significant
impact; however, below we note some reservations about continuing to
invest heavily in this area of research.

In concluding this section on the outcome goal of ideas, we summarize by
moving beyond substance to point to the practical contributions of these
ideas, their methodological contributions, and the ways in which the
research of the LSS Program has enriched disciplinary research.

The LSS Program does a superb job of funding studies that provide a
direct and clear service to a variety of social groups. Among the
studies with clear practical import, we include Evelyn Brodkin's
(0129643) research on the shift from public to private implementation of
welfare law. Her project was designed to provide data on the impact of
the Temporary Assistance Act to Needy Families in Chicago. Michelle
White's (9617712) study of financial distress and the use by debtors and
creditors of procedures such as bankruptcy, wage garnishment, and debt
renegotiation was designed to enhance our understanding of what can
happen to individuals and firms during periods of economic downturn. A
final example is David Kessler's (9709612) research on the impact of
legal and regulatory policy on patient and health provider
decision-making.

A hallmark of law and society research has been its empirical rigor. The
study of legal processes incorporates a range of methods as broad as
those encompassing every social science discipline. As we discuss in
more detail in Section B.3 (Outcome Goal for Tools), the Law and Social
Science Program has funded the development of important research tools,
such as data sets, surveys, experimental methodologies, that will be
used by scholars conducting future research. In addition, qualitative
research has yielded interview data and detailed case studies useful in
developing new research. For example, methodological innovations in
ethnographic field research on law have been shared across the
disciplines in law and social science. This range of methods and their
rigorous application across the field has facilitated the sophisticated
substantive and theoretical contributions we pointed to above.

Finally, LSS Program research contributes to the development of the
social sciences generally. LSS studies often offer new perspectives on
established disciplinary scholarship by importing existing theories into
the study of law and testing these theories in the legal arena. In other
instances, core social processes can be studied especially well in the
legal arena and theoretical innovations can then be exported to the main
social science disciplines. As an example of this latter pattern, we
point to research on regulation and institutionalization. LSS-funded
empirical studies of institutionalization are every bit as important to
the development of institutional theory in sociology and political
science as they are to the understanding of legal processes. Moreover,
the development of such theory in the interdisciplinary LSS environment
helps break down disciplinary balkanization and so speeds scientific
progress. As evidence of this synergy, we point to the significant
number of proposals that are jointly considered and jointly funded by
LSS and disciplinary programs in the social sciences. We would urge
further collaboration among the disciplines, noting that some (e.g.,
political science rarely co-review or co-sponsor proposals.

[Shifts in Funding Priorities]{.underline}:

We noted that in the declined proposals in the perception and cognition
and jury study sub-areas of psychology and law, reviewers raise concerns
about the overall research program. Some of the unsuccessful proposals
were characterized by reviewers as merely offering small iterations on
the well-established conclusions of previous research efforts. While
these proposed studies were generally well designed, they did not seem
to promise much return on the investment. Given the development of the
fields, they seemed unlikely to yield much theory-building or
methodological innovation.

In declining one proposal, for example, panelists acknowledged that the
researcher proposed appropriate methods and analytic procedures.
However, they concluded that \"the research would add incrementally in a
field about which a great deal has been explored.\" In those areas where
some of the main findings have already been thoroughly established,
perhaps potential innovation rather than excellence in project design
should govern the ultimate decision to fund. The Program Officer should
find a way to communicate this subtle shift in criteria to the central
figures in the field so as to discourage new proposals of the \"next
iteration\" type. These studies warrant special concern as the requested
amounts are generally quite large. Funding several such studies in one
proposal round can diminish the budget significantly; because of the
opportunity costs of funding multiple large proposals, careful
consideration is warranted. In addition, because the Program tends to
receive multiple proposals in these areas in each round, the labor
required of reviewers and panelists is considerable.

*B.3 --- Outcome Goal for Tools*

Over the past few years, the Program has energetically supported the
construction of major databases that have produced and promise to
continue to produce significant advances in knowledge. These databases
create critically important foundations for empirical research that join
the interests of scholars in a common enterprise while fostering
ancillary research into more detailed aspects of legal institutions and
socio-legal phenomena that would not be feasible without the advantage
these infrastructures provide scholars in starting their projects.

One of the first infrastructure awards supported the Supreme Court
Database, a coded compilation of the Court's decisions since 1948 that
includes information on the substance of the decisions, the votes of the
justices, and related aspects of the cases. This database has been
updated to include the Court's decisions to grant judicial review,
modified to allow for individual rather than only case-level analyses
(Spaeth 9910535) and more recently on the "relisting" of cases (Spaeth
0211617). This database has contributed substantially to the advance of
the systematic, empirical study of the Supreme Court through numerous
publications. It is the chief source of valid, reliable data on trends
in decision outcomes, voting patterns, and other aspects of the Court. A
web-based version of the database enhances the accessibility of the
database to the Program's research communities. A related but separate
project (Kauffman 0213999) is digitizing the records and briefs from 300
of the Court's most important decisions. These records will be placed on
the project's website to provide public access to the records.

A second major database traces the decisions of the U.S. Courts of
Appeals from the 1920s through 1988 (Songer). It rests on samples of
each of the circuits' published decisions over these seventy-five years
and, after necessary modifications, parallels the inventory of relevant
variables found in the Supreme Court Database. This database
systematically brings into light new information on these lower courts
that was hitherto unavailable. This database encourages longitudinal
analyses of the appeals courts as well as cross-sectional studies. This
database in turn is complemented by another database on the social and
economic backgrounds of the judges (Gryski 0119068). (It warrants noting
that the Program recently awarded a proposal to create a similar
database on confirmed and non-confirmed nominees to the U.S. Supreme
Court (Epstein 0240764).) The importance of the U.S. Courts of Appeals
database rests in the fact that it encourages judicial scholars to
explore the lower courts rather than concentrating their attentions
exclusively on the Supreme Court. Without this resource, research would
be episodic, less systematic, and consequently non-cumulative.

Similar problems have plagued the study of state supreme courts. For
this reason, the Program's decision to fund a third collaborative
infrastructure project (Brace 9616891, Hall 9911166) that focuses on the
fifty state supreme courts marks a major advance. Scholars have long
neglected these courts and this database creates opportunities for a new
wave of research that tests the generality of findings that rest on
research of the Supreme Court. Like the other databases, this database
is available through a web version that permits the downloading of data
for individual analyses. Finally, the Program has awarded an ambitious
comparative, transnational project (Tate 9975315) that is collecting
data on the high court decisions in over a dozen countries. This
database also allows for the replication of findings in the United
States to test their generality and thus broaden the empirical grounds
for theory construction. This transnational project is joined by many
other comparative studies of courts and law in other countries funded by
the Program.

These infrastructure projects are a unique construction by the Program
and contribution to the knowledge and understanding of law and law-like
institutions; no other nation has this wealth of systematically
collected, valid, and reliable data available to citizens and scholars
alike with which to explore the functioning and behavior of American and
foreign courts. It is a unique achievement.

*B.4 --- Outcome Goal for Organizational Excellence*

The LSS Program is structured in ways that encourage responsiveness and
efficiency in its relationship with its research constituencies. The
reliance of the review process on leading, productive scholars assures
that awards will reflect the interests and emerging concerns of the
Program's constituencies as well as further developments in their
research methodologies. Rotation of the Program Officer also can
contribute to the infusion of new ideas and perspectives while assuring
efficient accountability. The current Program Officer's attendance at
scholarly meetings is an especially important strategy toward this goal.
Last but not least, NSF's Fastlane has speeded up the process of
proposal submission as well as the review process and has reduced "dwell
time" for proposals as well as the "transaction costs" of both
processes. Finally, it bears noting that Fastlane has improved the
transparency of the Program's procedures and decisions.

##### NATIONAL SCIENCE FOUNDATION

##### POLITICAL SCIENCE PROGRAM, COMMITTEE OF VISITORS

##### FY 2004 REPORT 

+-----------------------------------------------------------------------+
| ##### Date of COV: March 18-20, 2004                                  |
+-----------------------------------------------------------------------+
| **Program/Cluster: Political Science**                                |
+-----------------------------------------------------------------------+
| **Division: Social and Economic Sciences (SES)**                      |
+-----------------------------------------------------------------------+
| **Directorate: Social, Behavioral, and Economic Science (SBE)**       |
+-----------------------------------------------------------------------+
| **Number of actions reviewed by COV: Awards: 75 Declinations: 65      |
| Other:**                                                              |
+-----------------------------------------------------------------------+
| **Total number of actions within Program/Cluster/Division during      |
| period being reviewed by COV: Awards: 176 Declinations: 460 Other:    |
| 10**                                                                  |
+-----------------------------------------------------------------------+
| **Manner in which reviewed actions were selected: Random Sample**     |
+-----------------------------------------------------------------------+

NOTE: **The COV elected first to provide our responses to Part B of the
questionnaire; this is below and entitled "Review of Program Results."
Following this we provided our detailed answers to questions A1 through
A5 in prose; then the answers to Part C. An appendix includes a list of
detailed answers to the specific questions posed in the form.**

#####  PART B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
the first three** **(People, Ideas and Tools)** **questions** **in**
**this section are to be based on the COV's study of award results,
which are direct and indirect accomplishments of projects supported by
the program. These projects may be currently active or closed out during
the previous three fiscal years. The COV review may also include
consideration of significant impacts and advances that have developed
since the previous COV review and are demonstrably linked to NSF
investments, regardless of when the investments were made. Incremental
progress made on results reported in prior fiscal years may also be
considered.**

The following questions are developed using the NSF outcome goals in the
NSF Strategic Plan. The COV should look carefully at and comment on (1)
noteworthy achievements of the year based on NSF awards; (2) the ways in
which funded projects have collectively affected progress toward NSF's
mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

Excellence in managing NSF underpins all of the agency's activities. For
the response to the Outcome Goal for Organizational Excellence, the COV
should comment, where appropriate, on NSF providing an agile, innovative
organization. Critical indicators in this area include (1) operation of
a credible, efficient merit review system; (2) utilizing and sustaining
broad access to new and emerging technologies for business application;
(3) developing a diverse, capable, motivated staff that operates with
efficiency and integrity; and (4) developing and using performance
assessment tools and measures to provide an environment of continuous
improvement in NSF's intellectual investments as well as its management
effectiveness.

NOTE: The COV has provided a general answer to these questions with the
following essay. An appendix includes specific answers to the detailed
questions posed.

# **National Science Foundation**

[Political Science Program]{.underline}

[FY 2004 Committee of Visitors]{.underline}

# **Review of Program Results**

**[Overview:]{.underline}**

We live in a world where combating terrorism, understanding the ethnic
and religious roots of political mobilization, advancing democracy, and
improving relationships with and among other countries have become
increasingly important for the safety and security of Americans.[^1]
Economic globalization also means that we must understand the political
forces that lead to flows of capital and flows of people. Domestically,
we must deal with major questions about the design of welfare programs,
campaign finance laws, and our political institutions. The National
Science Foundation's Political Science Program funds an extraordinary
array of research projects that address these issues. A sampling of
funded projects from 2001-2003 suggests the breadth and importance of
the topics. Our review of this sample indicates that NSF funded research
is characterized by the development of models and the systematic
collection of data to understand and test theories of political behavior
across all these topics. Through this application of the scientific
method NSF funded research develops fundamental knowledge about
political institutions, politics, and behavior that is vital for
American policymakers at all levels.

In American politics, for example, the National Science Foundation has
funded research on the effects of external threats (such as 9/11) on
U.S. domestic and foreign policy attitudes (Marcus; Huddy), the impacts
of the Internet on political participation and information seeking
(Valentino), distinguishing principles from prejudice in American's
views of racial policy (Huddy and Feldman), and the role of emotion in
mass political judgments (Lodge). It has also funded research on the
interaction between Congressional committee markups and floor votes
(Londregan), the efficacy of legislative oversight (Milyo), and the way
that legislators represent the policy preferences of their constituents
(Katznelson and Lapinski).

In public policy, the NSF political science program has funded the
development of comparative welfare state data (Scruggs), the impact of
campaign finance laws in state legislative elections (Hamm), the impact
of terms limits (Niemi), the capacity of school choice for enhancing
citizen participation (Schneider), as well as a broad
infrastructure-based policy agendas project (Jones and Baumgartner)
discussed below.

In international relations, projects have been funded to study the
design of reliable international agreements (Leeds), the dynamics of
international mediation processes (Gerner), the efficacy of laws of war
(Morrow), the empirical testing of crisis bargaining models with
incomplete information (Schultz), the impact of leadership turnover and
regime type on the evolution and maintenance of international
cooperation (Smith), and formal models of threats and economic sanctions
(Morgan). Projects have also been funded that have studied mass killing
and resistance in the Rwandan genocide (Davenport), military strategy
and civilian suffering in wartime (Huth), and the flows of refugees and
those are who displaced within countries (Moore). A project has also
created a data set for human rights research (Cingranelli). In
international political economy, projects have been supported on the
political economy of exchange rate regimes (Leblang and Willett),
economic insecurity and the globalization of production (Scheve and
Slaughter), and regional trade agreements as security institutions
(Powers).

In comparative politics, NSF has funded studies of ethno-nationalism and
electoral extremism (Kopstein and Wittenberg), regime preference and
political values in Jordan, Kuwait, and Palestine (Tessler), the
relative impact of Islam or modernization on the clash of civilizations
(Inglehart), formal and empirical approaches to ethnic mobilization
(Laitin), agent-based models of in-group favoritism and out-group
hostility (Axelrod), and the politics of identity and democratic values
(Sniderman). It has also funded work on the development of post Soviet
party systems (Miller), the comparative study of electoral systems
(Shively), strategic voting in the Israeli Knesset (Aldrich), cabinet
turnover in parliamentary democracies (Huber), measures of democratic
performance in Western democracies (Kim), and coalition government and
parliamentary oversight (Martin and Vanberg).

While speaking to the breadth and diversity of the program's
intellectual activities, this concise overview does not provide detail
on any one of these important projects. Let us take one example,
therefore, that illustrates the impact of an innovative Political
Science Program initiative, the EITM initiative.

[Empirical Implications of Theoretical Models (EITM)]{.underline}

Perhaps the most exciting initiative of the political science program
has been the formulation and implementation of the "EITM" idea that
calls for training and research in the Empirical Implications of
Theoretical Models. One of the marks of a maturing science is the
increasing sophistication of its efforts to create theoretical models
and the ongoing development of empirical methods for measurement,
description, and testing. Both trends have occurred in political science
in the last thirty years. Formal modeling has become increasingly
sophisticated, adding to our understanding of the operation of
legislatures, electoral systems, international relations, mass political
behavior, and regime change. At the same time, political methodology has
developed its own quantitative and qualitative methods for measurement,
description, and testing. As in other sciences such as physics and
biology, however, where formal theory and empirical methodology have
sometimes developed separately, there has not always been enough contact
between theory and methodology in political science. There are too many
statistical methodologists who think that a regression equation or a
likelihood function constitutes a model. And there are too many formal
modelers who fail to test their models.

EITM was born out of the concerns of a group of distinguished political
scientists, both modelers and empiricists, who met in July 2001 at the
behest of NSF's Political Science Program. The workshop concluded that
the separate development of formal theory and political methodology had
allowed each group to develop its own strengths and perspectives, but
the time had come to bring the two together, "bridging the divide
between formal and empirical modeling" by specifying models and devising
methodologically rigorous tests of them. The workshop recommended three
steps: education (training and retraining), dissemination of knowledge
(conferences and workshops), and research.

The education initiative led to grants for the creation of two
month-long summer training programs -- one at Washington University and
the other at (successively) Harvard, Michigan, Duke, and Berkeley. Each
program will train approximately 25 students yearly for four years for a
total of 200 students for the two programs. Impressively, the demand for
the program has exceeded expectations and the total number of applicants
has been about twice as large as projected. Faculty members from leading
institutions in the U.S. have been invited to lead seminars and to
provide mentoring opportunities for institute attendees.

In addition to the training programs for junior scholars, a number of
senior scholars have been partially supported to allow them to take
additional courses to address skill deficits in the areas of formal
modeling and empirical methodology. The result of these efforts has been
the rapid development of a community of scholars who are committed to
the EITM effort. Already, "EITM" has become an important phrase within
the discipline, and it is palpably changing the nature of research
undertaken by junior and senior scholars alike.

The dissemination of knowledge initiative has led to at least two
faculty workshops on "Political Institutions and Economic Policy" (at
Harvard University) and "Formal and Empirical Approaches to Ethnic
Mobilization" (at Stanford University). There has also been support for
a Research Work Team (at Indiana University) on the formal and empirical
analysis of repeated social dilemma games. As an example of the impact
and value of these workshops, the \"Political Institutions and Economic
Policy\" workshops run twice yearly at Harvard University by Jeffry
Frieden and Kenneth Shepsle bring together, repeatedly, leading scholars
from two communities. Political institutions scholars carefully model
the operating characteristics of institutional arrangements, in single
countries or comparatively, but there is often no "economy" in their
institutional world. Students of economic policy, on the other hand,
focus on patterns of economic growth and development, trade, investment,
etc., but there is typically no political institution beyond the
occasional median voter in their work. The workshop topics and papers
attempt to arbitrage across this divide, bringing to the institutions
scholars a more explicit sensitivity to the arguments of the utility
functions of agents in their model institutions, and bringing to the
economic policy scholars a greater appreciation for the impact of
institutions---legislatures, bureaucracies, courts, and elections
particularly---on the selection and implementation of policy. In sum,
these workshops are exercises in intellectual arbitrage and
cross-fertilization.

Several individual research awards already have been made under EITM
within the Political Science Program. In 2003, EITM was made an explicit
component (using the EITM name) of the Human and Social Dynamics
priority area for SBE with a special competition of one million dollars.
In 2004, EITM was incorporated into the Human and Social Dynamics
priority area through language that mentioned the core EITM concepts. We
are very pleased that a political science initiative has become part of
the larger HSD priority area, and we expect that some excellent
political science will be funded through this initiative. These positive
developments make possible a far greater research impact of the EITM
idea in the future. At the same time, we believe that it would be even
more useful to provide an increase to the political science program
budget that could be devoted to EITM in order to ensure that the
promising beginnings of EITM -- which were funded through the political
science program budget -- are realized. Political science is at a point
where the EITM initiative is having an especially large impact. The
political science program had the vision to develop the program, and it
invested the start-up funds to demonstrate its feasibility. We believe
that such innovation should be rewarded, especially given the
substantial benefits that will be realized within political science by
expanding the resources available for the EITM initiative.

**[Linking teaching and research; creating decentralized research
platforms:]{.underline}**

One of the most important contributions of NSF-sponsored research is the
creation of research platforms and the dissemination of research results
and tools that can be used by large numbers of scholars. The money spent
by NSF on these projects is successfully leveraged to produce many times
the return on the original cost of the project. NSF-supported projects
have adopted innovative technologies for data distribution and analysis
as well. For example, the Policy Agendas Project
([www.policyagendas.org](http://www.policyagendas.org/)) not only allows
for the distribution of historical data on congressional hearings,
statutes, the federal budget, and other data resources concerning
government activities in the post-1947 period, but its interactive web
site also allows students and scholars to analyze data interactively.
Such technologies allow not only larger teams of collaborators to work
together on research projects, but also provide the opportunity to
support a wide range of individual research projects across many areas
of congressional studies, public administration, and public policy.
Thus, a single infrastructure project has the potential to support a
number of research advances. In addition, the data and web-based
distribution and analysis tools are useful for teaching as well as
research projects.

Similarly, the Correlates of War-2 and the EUGene software program have
become central elements in the IR research program across the country,
used by scholars in many areas and supporting research on international
conflict and in comparative politics (through the Minorities at Risk
program and the related MARGene software adaptation). These large
projects have several meritorious characteristics: they are high quality
science projects on their own; they enhance the infrastructure of
science; they allow a variety of scholars to use common data resources
including those working in universities with fewer research facilities
to compete intellectually with those at resource-rich universities; and
they can have a direct and substantial impact on undergraduate teaching
as well.

The Political Science program has invested heavily over the years in
programs designed to enhance the training of Political Scientists. For
example, the Program supports a large number of Doctoral Dissertation
Improvement grants enabling some of the best and brightest young
scientists in the discipline to substantially increase the quality of
the research they are able to undertake as students. These awards have
proven especially useful for providing graduate students the resources
needed to undertake field research, collect original data, pay
experimental costs, and otherwise increase the empirical significance of
their dissertation research.

The Political Methodology Summer Workshop is another longstanding
program, started and sustained by NSF funding, that continues to pay
important dividends. The workshop serves both to identify and develop
some of the best young quantitative methodologists in the discipline. In
recent years, at NSF's urging, the Workshop has given increased
attention to the identification and development of women and minority
Political Scientists.

Indeed, the Political Science Program has been especially committed to
increasing the training of traditionally under-represented groups
including women and minority Political Scientists. A good example in
this regard in the Program's long support for the Ralph Bunche Institute
which has an unparalleled record of recruiting and training young
African-American, Hispanic and Native American Political Scientists.

Given recent advances in qualitative research methods, the Program has
also provided recent support for a winter Training Institute on
Qualitative Research Methods at Arizona State University. The Institute
brings together approximately 60 graduate students and junior faculty
and fifteen instructors for two weeks of intensive instruction in, and
discussion of, state of the qualitative research methods.

Although considerably less visible than these special purpose training
programs, some of the most effective graduate training supported by the
NSF Political Science Program occurs as a result of the substantial
funding of graduate research assistants that is a normal part of
virtually every research grant awarded by political science. The close
mentoring of small numbers of graduate students by an experienced
principal Investigators working on an NSF supported research project
remains one of the most effective means of training young scientists of
which we are aware.

[ANES:]{.underline}

The American National Election Studies (ANES) is one of the most
venerable and widely utilized data sets in the social sciences. Like
those just mentioned, but moreso than any other project in the
profession, this resource has affected intellectual trends, created
decentralized research opportunities for scholars, and affected teaching
about political science in the classroom. The election studies are
random sample surveys of the American public extending back to 1948,
providing an invaluable time series for understanding elections,
campaigns, and political behavior. The PIs of the ANES have done an
outstanding job of guaranteeing data accessibility, through the website
at [www.umich.edu/\~nes](http://www.umich.edu/~nes), or on CD ROM.
Because the data are so easily accessible, they are extremely important
in the education of undergraduates, the training and dissertation
research projects of graduate students, and in the research programs of
junior scholars and senior scholars, including those with no involvement
in the development of the instrument itself.

The Political Science Program has been an effective steward of the ANES,
and has been proactive in insuring that the election studies continue to
adhere to the highest scientific standards. Following the National
Science Board Resolution on Recompetition in 1997 and the 1998 COV
report that recommended recompetition, the Program held a recompetition
for the ANES in 2000 which resulted in an award to the University of
Michigan for the 2004 election study. Recompetitions for future election
studies are also planned. In addition, the program sponsored an American
Electoral Behavior Workshop in 2003, where participants provided advice
about the methodological and substantive challenges that are faced by
the ANES and NSF as survey costs escalate faster than NSF budgets. Among
the most important tradeoffs to be faced involve maintaining the time
series "core" of the ANES versus providing the capacity for pilot
research on innovative methodologies and theories. In addition, the ANES
relies on face-to-face interviewing, a methodologically desirable but
increasingly expensive research strategy. The ANES Board of Overseers
has not developed a strategy for a possible transition to more
cost-effective modes of data collections, despite NSF efforts to
encourage it to do so. The ANES is an important project of undisputed
scientific integrity, and the Political Science Program is to be
commended for its continued support. At the same time, the Program
should continue its strategy of re-evaluating the methodological,
theoretical and substantive challenges, as well as the financial
implications of this large commitment.

[TESS:]{.underline}

Time-sharing Experiments for the Social Sciences (TESS) is a much newer
NSF initiative. The result of an Infrastructure Award to Diana Mutz and
Arthur Lupia, TESS is a platform that provides social scientists with
opportunities for original data collection. TESS provides two
large-scale, ongoing, and cooperative data collection opportunities, one
involving a national telephone survey, the other an Internet-based
survey. Scholars compete for time on one of these instruments, through a
comprehensive review process that screens proposals for the importance
of their scientific contributions. The review process is overseen by the
co-PIs and a group of over 45 Associate PIs, prominent scholars who
represent the wide range of social science disciplines. TESS has several
innovative aspects. The first is that it combines the strengths of
national random samples (that is, the ability to generalize results)
with the strengths of randomized experiments (that is, the ability to
reach strong and confident causal conclusions). As a result, scholars
are able to increase the precision with which fundamental social and
political dynamics are measured and understood. Second, TESS is
multidisciplinary, available to scholars from all the social sciences.
Third, TESS maximizes financial efficiency, by distributing the costs of
sampling, interviewing and administration over a large number of
studies, thereby reducing the marginal cost of each study substantially.
The annual reports submitted by the co-PIs indicate that TESS has been
successful in accomplishing its stated goals. As of February 2004, 178
proposals have been received; the submitting scholars are from multiple
fields, levels (senior, junior, and graduate students), types of
institutions, and geographic areas.

**[Recommendations:]{.underline}**

We hesitate to propose significant changes to a program that is
functioning well and serving the scientific community consistently.
However, there are a number of issues relating to trends in the
discipline that are worth serious consideration by the Foundation and
its officers. We note just a few, though others are apparent from our
comments throughout this document:

-   Our most important recommendation is that the Program be recognized
    and rewarded for its success with the EITM initiative. In our view
    this is not only innovative in the best sense but it has the
    potential to affect on-going research, graduate training, and the
    future of the profession. We urge NSF to provide a permanent
    increase in the Political Science budget earmarked specifically to
    fund research and training related to and building on the EITM
    model, and we further urge the Foundation to ensure that political
    scientists are integrated into other EITM-related activities in the
    Foundation.

-   Non-dissertation fellowship money (like that in EITM) is needed to
    allow students extra time to take methodological and formal
    theoretical course work at their home universities, post-general
    exams. NSF must take a stand on finding the resources to allow
    students to gain the needed advanced training in more than one area
    and more than one methodology. In a truly integrated profession,
    more skills are needed; these are costly.

-   NSF-Political Science should consider sponsoring a workshop, joint
    with other SBE programs, to think about experimental laboratory
    infrastructure needs in the next decade. Shared infrastructure
    relating to experimental work may be an important initiative for the
    Foundation to explore.

-   Reduce the tradeoffs between the number, duration, and size of
    grants---it is getting to the point where grants of under \$100 K
    have reduced impact on scientific progress. Average grant sizes are
    now below this level.  This is largely a matter of increasing core
    program budgets at NSF which have not kept up with the true cost of
    research in the SBE disciplines over the past twenty years.

-   Continue and expand support for infrastructure in NSF awards,
    whether through a formal infrastructure program or through attention
    to supporting projects that will produce shared and widely
    disseminated research tools, datasets, or platforms, especially
    through decentralized web access. Creating a culture of shared us of
    common data resources is an important long-term goal for the
    profession. It fosters wider collaborations and enhances the chances
    for scientific replication.

Although we applaud the Human and Social Dynamics initiative as a way to
support innovative and novel social science research, we worry that core
disciplinary program budgets are not being substantially increased and
do not appear to be a priority for future increases. Similarly, we worry
that NSF administrators are not addressing systematically the difficult
trade-offs and decisions that affect core programs. The path of least
resistance would be to hope that large budget increases over the next
several years allow no programs to be cut and to reward meritorious
programs and initiatives with differential growth out of the overall
funding increases. However, it may be more likely that the large budget
increases we hope for will not materialize.  In that case administrators
will need to make much more difficult choices in order to sustain and
expand core programs through reallocation of available resources.

PART A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT

A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures.

The members of the Committee of Visitors evaluated the quality and
effectiveness of the program's use of merit review procedures very
positively. The Political Science Program makes use of the standard
two-step review procedure consisting of both ad hoc and panel review.
For standard proposals supporting research by a single or small group of
scholars, the Program Officers solicit ad hoc reviews from scholars with
research interests relevant to the proposed research. Care is also taken
to solicit ad hoc reviews from scholars with relevant methodological
skills. Most reviewers are appropriately political scientists, but
reviewers from other disciplines are also used as well. In addition to
ad hoc reviewers, typically at least two reviews are written by members
of the panel. Program statistics indicate that an average of 6 reviewers
are solicited; decisions are typically based on 3-5 reviews. For big
budget and infrastructure proposals, more reviewers are solicited, and
decisions appropriately are based on a larger number of reviews.

For the period under review, dissertation proposals were also reviewed
at the regular meeting of the Political Science Panel. Those reviews
were based upon the reviews of three members of the panel. Because the
number of both standard and dissertation proposals has steadily
increased, the review burden on the panel members has increased, and so
the Program Officers have decided to create a separate dissertation
review panel that meets once a year. The members of the COV believe this
is a wise decision that will preserve both the integrity and efficiency
of the reviews of both types of proposals.

On rare occasions, the Program Officers make awards under the SGER
program. For example, subsequent to the Sept. 11 terrorist attacks, the
program made three SGER awards to study public attitudes; the members of
the COV believe such awards represent a judicious use of program funds
when time-constrained opportunities arise.

**The Program operates with an outstanding degree of efficiency. Those
who submit proposals are informed as rapidly as possible about the
Program's decisions, virtually all within six months of submission.
Given the complexity and thoroughness of the two-step review process, as
well as the inevitable requirements of the bureaucratic process, we view
this time to decision to be appropriate. However, as the number of
proposal submissions increases and the Division moves increasingly into
priority research areas that are outside of the normal Program
jurisdictions (but still require the time of regular Program staff), we
expect the demand on staff to become more pronounced and the
consequences more serious.**

We were impressed with the quality of the reviews included in the
Program jackets. The Program has been consistently successful in its
efforts to support scientific research that is theoretically motivated
and methodologically rigorous. Past productivity is a criterion taken
into account in proposal reviews. Importantly, senior scholars and
junior scholars alike are subject to rigorous and focused, detailed,
probing reviews. Indeed, we were impressed by the high levels of success
enjoyed by junior scholars. Senior scholars are not likely to be passed
along if they submit what may be called a "trust-me" proposal.
Previously rejected proposals, when revised and resubmitted, are
subjected to serious questions about how well the revised proposal rises
to the specific challenges posed by the previous panel and reviewers.
Continuity on the panel ensures that PIs hoping to "slide by" do not do
so. We saw several examples of very senior scholars being commended for
their previous accomplishments but (rightly) called to task for the lack
of specifics on a particular research question or for a fatal flaw in a
research plan. Perhaps most telling in this process is a review of
individual senior scholar's "hit rates" with NSF: those with the most
successful proposals also show a great number of declines in their
histories. This is no old-boy network; the review process appears to
differentiate between detailed and complete proposals from "trust-me"
proposals, even when the track record of a given PI might indicate that
some trust is warranted. It simply doesn't work that way.

The individual reviews and the panel summaries provide sufficient
information for the principal investigator(s) to understand the basis
for the Program's recommendation. Most PIs are provided with
constructive criticism of the proposed research. More generally, the
documentation of the review process leading up to the Program's
recommendation is very complete. We reviewed a large sample of jackets,
both successes and declinations, and were favorably impressed with the
detailed documentation of the review process. When disagreement between
the ad hoc and panel reviews is evident, the documentation provides a
clear rationale for the resulting funding recommendation.

Recommendations: Continue to seek out high quality reviews as has been
done in the past.

A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers.

Comments: Our review of a sample of jackets for the past three years
indicates that Political Science reviewers and panelists give careful
attention to the scientific merit of proposals. Substantial attention
focuses both on the theoretical innovations promised by a proposal and
on the quality and feasibility of the research design. Political Science
reviews typically give less explicit attention to the broader social
impacts of proposed research. In most cases, this is because a
consideration of social impacts is inherent in a consideration of
scientific merit. In Political Science the potential social significance
of the fundamental questions underlying our research provides the basic
motivation for undertaking the research in the first place.

While Political Science reviewers are not always careful to distinguish
scientific merit and broader social significance, a review of Panel
Summaries and Form 7's indicates that Program Officers are much more
diligent about separating the two criteria and addressing them
individually. The "nuggets" prepared by the Program to better inform the
Foundation, Congress and the general public regarding Program activities
also attend carefully and in detail to the broader social impacts of the
work Political Science funds.

In the Social, Behavioral and Economic Sciences, distinguishing
scientific merit and social impacts is frequently impossible. While
reviewers need to be encouraged to attend to this distinction where it
makes sense, the Foundation needs to be more flexible in recognizing
that the distinction often is artificial and cannot reasonably be made.

Recommendations: Although we believe that the current review process
gives ample consideration to both scientific merit and broader social
impacts, Political Science 'ad hoc' reviewers could do somewhat more to
separate their discussion of these closely related but potentially
separable evaluation criteria. As the Program Officers begin to
experiment with the use of more personalized cover letters to reviewers,
they may want to use this new opportunity to reinforce the message to
the reviewer community of the need to address both of the review
criteria separately. In general, however, we believe that the Program
has adequately addressed the two review criteria.

A.3 Questions concerning the selection of reviewers.

Finding an adequate number of reviewers has gotten increasingly
difficult. The refusal rate has grown, and constraints on who may review
during any given cycle have become increasingly binding. Nevertheless,
the Program Officers are to be commended for their diligence in seeking
out reviewers, and for using innovative methodologies for encouraging an
adequate response rate.

Every proposal jacket we examined obtained at least two reviews from
panelists. Non-dissertation proposals receive reviews from additional
readers, normally two to four. On average, just above six ad hoc reviews
were solicited and about three per proposal were returned. Dissertation
proposals ordinarily receive one additional reading. In one of the
cycles during this period, the program was inundated with such a large
number of dissertation proposals that a "virtual panel" of reviewers was
established, enabling each dissertation proposal to receive multiple
outside readings and a comparative perspective. This is a very
innovative development. We understand that the dissertation competition
is now once per year and an [actual]{.underline} panel has been
assembled exclusively for this purpose.

The reviews generally were of very high quality and, in most cases,
conveyed a consensus enabling the Program Officers to come to an
unequivocal judgment. This underscores the care brought by the Program
Officers in appropriately selecting reviewers.

Reviewers come primarily from research universities, and thus display
some variability in terms of geographic distribution. Primary emphasis
appears to be given to reviewer expertise.

**The Program Officers are very aware of conflict-of-interest issues,
some of which entail quite binding constraints on their choice of
reviewers. These involved the panel as well, and the jackets clearly
noted when panelists recused themselves from discussions of particular
proposals when conficts arose.**

Concerns relevant to selection of reviewers: Constraints on reviewers
have become increasingly binding. Response rates have fallen from well
above 50% to below 50%. The Program Officers may wish to consider
further experimentation with encouraging response.

There is some impressionistic evidence that ad hoc reviews are somewhat
shorter and less informative than in the past. This is not so much a
problem for award decision making, but does reduce the feedback to
proposal writers.

Recommendations: The use of a "virtual panel" for dissertation proposals
to reduce demands on the program panel, and to bring a comparative
evaluative dimension to the review process, was an excellent
development. The move to a permanent separate panel for this purpose
makes sense and we applaud it.

The Program Officers should continue their innovative experiments to
induce an increase in the reviewer response rate (personalized request
letters and "tickler systems" for example). They are also encouraged to
explore further the effects of "fast lane" effects on response rate.

We urge the Program Officers to continue to pay attention to racial and
gender diversity of the panels; we note that 5 of 19 panelists in recent
years have been female and two have been members of under-represented
racial groups. This level of representation reflects significant effort
on the part of the Program Officers; we urge them to continue with these
efforts.

A.4 Questions concerning the resulting portfolio of awards under
review**.**

The COV reviewed scores of proposal ranging from dissertations to
single-investigator awards to large infrastructure grants, including
both successful and unsuccessful proposals. Unanimously, we reached the
conclusion that the quality of science being proposed and the fairness,
balance, and the merit-based review process were extremely high. In
particular, we found a great unwillingness of panels and Program
Officers to support "trust-me" proposals: even the most well qualified
and senior scholars are held to a standard that requires them to
elucidate, in detail, their plans, operationalizations, and expected
outcomes. We found this to be an especially strong indicator of the
quality of the science in the discipline (since to be competitive in
this field one must reach a high degree of specificity in the proposal,
in addition to having innovative ideas) as well as a sign of a healthy
review process based on merit. The number of well-known senior figures
with unsuccessful grant proposals was striking. Many senior scholars had
track records consisting of several unsuccessful proposals as well as
some successful ones. We also saw as a healthy sign of the review
process that many PI's come back to the panel with a subsequent proposal
after being denied funding in one round. Generally, the feedback they
receive from ad-hoc reviewers, from panelist reviewers, and from the
panel and Program Officers is substantive and detailed, allowing the PI
to judge whether a revision is feasible or if the project is not likely
to be funded even if revised. Since many do indeed come back with
successful proposals, we think that the review process is generating
improvements in the proposals and that the scientific merit of the
process is strong.

Another sign of the strength of the review process in place, and the
confidence it elicits in the scientific community, is the pattern we
noted again and again where senior and junior scholars alike would be
unsuccessful in certain proposals but would return some years later to
the Foundation with another proposal on a different topic. Clearly,
scholars are not systematically "turned off" from seeking out NSF
funding for those projects where it may be appropriate in spite of
negative reviews in many cases. This seems a strong sign of the success
of the merit review process and of the feedback that scholars are
getting in the process. Many scholars have a mixed track record; many of
the most prominent have more misses than hits in fact, but they continue
to return to the Foundation with those proposals that they believe may
merit funding. We noted no one, in fact, who had been successful every
time they approached the Foundation. (It is also fair to say we did
observe a few who returned almost year after year especially with
dissertation proposals, with no success. However, the overall number of
those proposing consistently uncompetitive proposals was very low.) In
all, we had a strong sense of a review process that not only functions
well but also elicits strong support and respect in the discipline.

The mix of supported projects is good. However, we note with some
concern that overall budget constraints have led to a decline in the
average size and duration of awards in the past three years. From 2001
to 2003, the average size of an award declined from \$174k to only \$87k
(whereas the number of awards moved from 46 to 60). It is entirely
appropriate for the mix of proposals to change over the years, and we
appreciate the problem of reducing the number of awards in order to
provide greater support to each award funded. However, the relatively
small size of many of the awards means that their impact is small,
especially considering the time-consuming review and application
process. Many meritorious projects are not being funded because of
budget constraints. We feel strongly that the quality of science
supported would not be detrimentally affected if the size of the
Political Science Program's budget were increased substantially, even
doubled in a single year. An enlarged budget would lead to larger sized
individual awards, more awards, and perhaps most importantly a different
type of proposal. We note the high cost of certain types of research,
personal interviews in surveys for example. Where appropriate, surveys
based on personal interviews in a national probability sample can be
extremely expensive, easily leading to overall project costs in the
millions. Given current budget levels, the Program simply cannot support
many such proposals, but such a conclusion is strongly affected by
budget constraints, not only scientific criteria. These are, of course,
inevitable and we do not propose a blank check for the Program. However,
we note that the level of science apparent here could support with
dramatically increased financial resources; there would be improvement,
not dilution of quality, with more resources.

**The mix of types of projects supported (centers, individual proposals,
infrastructure awards...) is good. Several sub-field specific workshops
have been very successful in the past and we encourage the Program to
continue to support these. The impact of the investments of the Program
in methodology and in the EITM summer training institutes appear to be
particularly great, and we encourage more initiatives along these lines.
The impact of the program on graduate training and education, as well as
on methodological training for younger scholars after the PhD can be
great.**

In terms of sub-field specific issues outside the area of methodological
training, we believe that the subfields themselves should take these
initiatives. Program support should be dependent upon scientific merit
and broader social impacts. The NSF Political Science Program has been
most effective in helping deal with issues of graduate training in
particular, and we encourage even more attention to this in the future.
The initiatives in supporting large infrastructure projects have been
extremely important; in sum, the mix of types of proposals is
appropriate.

In terms of the diversity of principal investigators being supported, we
note as previous COVs have also noted that program statistics make it
difficult to assess the diversity of the pool, for two reasons. First is
that statistics are based on PI's and many proposals have a number of
Co-PI's or other senior personnel, but the statistics ignore these
personnel, many of whom may be women or minorities. Second,
self-reporting rates of these descriptive data are low and make it
difficult to interpret.

Concerns relevant to the quality of the projects or the balance of the
portfolio: We strongly believe that the review process is effective in
identifying the most meritorious proposals for support. For the most
part, the mix of projects supported has been appropriate and has been
reflective of the interests and proposals that come from the scientific
community. The Program has been effective in using initiatives, SGERs,
and special competitions to great effect. We are especially impressed
with the EITM initiative and see it as a model for the integration of
serious work on theoretical modeling with substantial empirical testing
as well. We commend the Program officers for their efforts to establish
large infrastructure awards as well as to continue to support
traditional investigator-based awards.

**While there is substantial focus on graduate training, we note that a
great deal of educational impact at both the graduate and undergraduate
levels occurs through the normal grants process. NSF-based statistics
may not always reflect the true impact of the Political Science Program
on education and training because much of this impact comes through the
functioning of individual research awards that incorporate graduate and
undergraduate students. We note the Ralph Bunche Institute as a
significant counter-example to this and as the only REU-Site project in
the Political Science area; we commend this project and note its unique
contribution in political science.**

Recommendations: The merit-review process is functioning very well.

Overall budget constraints are making it difficult to support very many
large projects, or even to support the existing portfolio of projects to
the level that would be beneficial. However, we commend the Program
Officers for making sure that the highest quality proposals are
supported and that PI's receive substantial feedback on their ideas,
allowing many of them to return with improved proposals that often
achieve funding after an initial rejection. (We also note that
continuity on the panel helps ensure that revised proposals that do not
address the scientific issues raised in earlier reviews tend to be
unsuccessful; there is little expected value in PI's simply hoping for
"better luck" in a subsequent review process; this tells us that the
merit review mechanisms in place are based on solid criteria of
scientific merit.) The Program officers are to be commended for their
initiatives and flexibility in designing review processes, including
special panels where appropriate, that allow each proposal to be
reviewed completely and fairly, but while avoiding such a multiplication
of review panels that none would have a broad view of the relative
merits of given proposals as compared to the overall mix of proposals.

In summary, we strongly believe that the merit review process and the
selection of proposals for support are handled very well. The Program
Officers are to be commended.

A.5 Management of the program under review**.**

A. Management of the program.

> **Political Science enjoys a well deserved reputation as one of the
> best managed programs at NSF. Program Officers, Frank Scioli and Jim
> Granato do a superb job of insuring that all proposal submitted to
> Political Science review a full and fair review, a timely decision,
> and extensive feedback. That they are able to do so despite increasing
> proposal pressure and the need to manage increasing numbers of
> initiatives and special competitions is all the more impressive and
> worthy of recognition and reward by the Foundation.**

B. Responsiveness of the program to emerging research and education
trends.

> **The Program goes to considerable lengths to insure that it is
> sensitive to emerging research and education trends. Program Officers
> regularly attend both regular professional meetings and special
> purpose conferences in order to track emerging theoretical and
> methodological trends. Program Officers also regularly set aside time
> at each panel meeting to discuss research trends with panelists, who
> include some of the leading scholars in the profession. The Program
> has been attentive to emerging educational trends, as reflected in its
> initiatives in EITM, political methodology, and qualitative methods.
> The Program has also made a recent decision to have a separate panel
> consider doctoral dissertation proposals that should provide
> additional opportunities and impetus for the Program to consider
> educational issues.**

C. Program planning and prioritization process (internal and external)
that guided the development of the portfolio under review.

> **The Program has been extremely effective in the planning and
> prioritization process. The Program's leadership in the Empirical
> Implication of Theoretical Models (EITM) initiative is the most
> visible and outstanding example in this regard. The initiative
> promises to revolutionize the way future generations of Political
> Scientists conduct research. The Program's role in the National
> Election Study recompetition is additional evidence in this same
> regard. Political Science Program Officers are to be commended for
> attending so diligently to planning activities despite the increasing
> workload they confront.**

D. Discuss any concerns identified that are relevant to the management
of the program.

> **The members of the COV are concerned principally about the rapidly
> increasing burdens being placed on the time and energy of Program
> Officers. In addition to continuing increases in proposal pressure,
> Program Officers are being asked to take responsibility for increasing
> numbers of cross-program initiatives and virtual programs. This not
> only increases the number of proposals Program Officers must process
> but also detracts from the ability of NSF staff to engage in planning,
> keep up with developments in the discipline and otherwise continue to
> manage the Program in an efficient and effective manner. While we want
> to emphasize that we have not seen any evidence as yet that the
> quality of the program has been compromised, we do not believe that
> NSF can continue to rely upon the heroic efforts of Program Officers
> to compensate for the lack of adequate Foundation staffing for the
> rapidly increasing initiatives and special programs.**

E. Recommendations:

> NSF needs to insure that new funding initiatives are supported by
> appropriate increases in staff so that existing Program Officers are
> able to continue devoting the time and energy necessary to maintain
> high quality core programs. The current Program Officers should be
> commended for their work.

##### PART C. OTHER TOPICS

\[Note: the COV covered these topics in our comments in Parts A and B,
above\]

C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.

**\[covered in comments above\]**

C.2 Please provide comments as appropriate on the program's performance
in meeting program-specific goals and objectives that are not covered by
the above questions.

**\[we addressed a number of issues especially in our comments in Part
B\]**

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

**\[addressed in comments above\]**

C.4 Please provide comments on any other issues the COV feels are
relevant.

**\[we addressed what we felt important to address in Part B\]**

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

\[review process allowed us ample access to needed information; this was
very well done. The report template was less helpful but it had the
merit of providing a clear and wide-ranging set of question which we
have attempted to address systematically in each section.\]

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the Political Science Committee of Visitors

Frank R. Baumgartner

**Chair**

Appendix to Part A: Answers to detailed questions relating to the
questions above.

A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures.

+-----------------------------------------------------------+----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES**  | **YES,   |
|                                                           | NO,**    |
|                                                           |          |
|                                                           | **DATA   |
|                                                           | NOT      |
|                                                           | AV       |
|                                                           | AILABLE, |
|                                                           | or NOT   |
|                                                           | APPL     |
|                                                           | ICABLE** |
+-----------------------------------------------------------+----------+
| Is the review mechanism appropriate? (panels, ad hoc      | Yes      |
| reviews, site visits)                                     |          |
+-----------------------------------------------------------+----------+
| Is the review process efficient and effective?            | Yes      |
+-----------------------------------------------------------+----------+
| Are reviews consistent with priorities and criteria       | Yes      |
| stated in the program's solicitations, announcements, and |          |
| guidelines?                                               |          |
+-----------------------------------------------------------+----------+
| Do the individual reviews (either mail or panel) provide  | Yes      |
| sufficient information for the principal investigator(s)  |          |
| to understand the basis for the reviewer's                |          |
| recommendation?                                           |          |
+-----------------------------------------------------------+----------+
| Do the panel summaries provide sufficient information for | Yes      |
| the principal investigator(s) to understand the basis for |          |
| the panel recommendation?                                 |          |
+-----------------------------------------------------------+----------+
| Is the documentation for recommendations complete, and    | Yes      |
| does the program officer provide sufficient information   |          |
| and justification for her/his recommendation?             |          |
+-----------------------------------------------------------+----------+
| Is the time to decision appropriate?                      | Yes      |
+-----------------------------------------------------------+----------+

A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers.

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       | YES       |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     | YES       |
| proposal contributes to both merit review criteria?      |           |
+----------------------------------------------------------+-----------+
| Have the *review analyses* (Form 7s) addressed whether   | YES       |
| the proposal contributes to both merit review criteria?  |           |
+----------------------------------------------------------+-----------+

**A.3 Questions concerning the selection of reviewers.**

+---------------------------------------------------------+------------+
| Selection of Reviewers                                  | YES , NO,  |
|                                                         |            |
|                                                         | DATA NOT   |
|                                                         | AVAILABLE, |
|                                                         |            |
|                                                         | or NOT     |
|                                                         | APPLICABLE |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       | **YES**    |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| *See program statistics for help in addressing this     |            |
| question.*                                              |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            | **YES**    |
| appropriate expertise and/or qualifications?            |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    | **YES**    |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| *See program statistics for help in addressing          |            |
| geographic and institutional balance; information on    |            |
| underrepresented groups is not available.*              |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      | **YES**    |
| interest when appropriate?                              |            |
+---------------------------------------------------------+------------+

**A.4 Questions concerning the resulting portfolio of awards under
review**.

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     | ####          |
| projects supported by the program.                   | # Appropriate |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  | **            |
| scope of the projects?                               | Appropriate** |
|                                                      |               |
| *See program statistics for help in addressing this  |               |
| question.*                                           |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   High Risk Proposals?                             |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Multidisciplinary Proposals?                     |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Innovative Proposals?                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Funding for centers, groups and awards to        |               |
|     individuals?                                     |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Awards to new investigators?                     |               |
|                                                      |               |
| *See program statistics for help in addressing this  |               |
| question.*                                           |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Geographical distribution of Principal           |               |
|     Investigators?                                   |               |
|                                                      |               |
| *See program statistics for help in addressing this  |               |
| question.*                                           |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Institutional types?                             |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance of:                                          | Appropriate** |
|                                                      |               |
| -   Projects that integrate research and education?  |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | **            |
| balance:                                             | Appropriate** |
|                                                      |               |
| -   Across disciplines and subdisciplines of the     |               |
|     activity and of emerging opportunities?          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          | **            |
| participation of underrepresented groups?            | Appropriate** |
|                                                      |               |
| *See program statistics for help in addressing this  |               |
| question.*                                           |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      | **            |
| agency mission, relevant fields and other customer   | Appropriate** |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
+------------------------------------------------------+---------------+

**Appendix to Part B. Detailed questions:**

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

+-----------------------------------------------------------------------+
| **B.1** **[OUTCOME GOAL for PEOPLE]{.underline}: Developing "a        |
| diverse,** **competitive and globally engaged workforce of            |
| scientists, engineers, technologists** **and well-prepared            |
| citizens."**                                                          |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| See our general comments above. The Political Science Program has had |
| strong impacts in this area.                                          |
|                                                                       |
| Building human capital in Political Science is especially effective   |
| when done at an early stage in a scholar's career. Stipends for       |
| dissertations are one way in which the Program does this. Another     |
| possibility we urge would provide students, perhaps after their       |
| general examinations, with support to spend an additional semester or |
| year training in subjects beyond their own discipline, taking         |
| economics or statistics courses, for example. Since they often have   |
| little time pre-generals to take in-depth methodological training,    |
| this would afford them additional opportunity to do so.               |
+-----------------------------------------------------------------------+
| **B.2** **[OUTCOME GOAL for IDEAS]{.underline}: Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| See our comments above. The Political Science Program supports a wide |
| range if theoretical topics including a great many that address       |
| fundamental social problems as well as issues relating to basic       |
| scientific knowledge.                                                 |
+-----------------------------------------------------------------------+
| **B.3 [OUTCOME GOAL for TOOLS:]{.underline} Providing "broadly        |
| accessible, state-of-the-art S&E facilities, tools** **and other      |
| infrastructure that enable discovery, learning and innovation."**     |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| See our comments above. The Political Science Program has had a       |
| strong impact in this area, though, as we note above, there is great  |
| need for increased attention to infrastructure needs in the           |
| scientific community. This includes both shared data resources as     |
| well as equipment and laboratories.                                   |
|                                                                       |
| American political science is the most scientifically advanced in the |
| world, in no small measure owing to the commitment of NSF to research |
| in political science. A continuing presence by NSF in efforts at      |
| creating and maintaining infrastructure is essential.                 |
|                                                                       |
| The American National Election Study (ANES) constitutes a             |
| long-running time series, the core of which is a significant data     |
| series on which to track changes in the American electorate.          |
| Maintaining this series is important. However, in light of the        |
| growing expense of face-to-face interviews, research on alternative   |
| methodologies for maintaining reliability while transitioning to less |
| expensive modes is also important.                                    |
|                                                                       |
| Experimental methods are a growing part of the political scientist's  |
| tool kit. On-line experimentation (Time-Sharing Experiments in the    |
| Social Sciences, or TESS) is a significant infrastructure development |
| of recent years. Investment in experimental laboratories is something |
| on which the program may wish to be more pro-active, possibly in      |
| cooperation with other programs in the division.                      |
|                                                                       |
| Other infrastructure projects have similarly been successful and      |
| harbor the promise of further improvements in the structure and       |
| impact of science in the discipline; we encourage continued and       |
| expanded focus on these projects.                                     |
|                                                                       |
| Methodological and theoretical training constitutes an important part |
| of human capital accumulation for Political Science. Leading research |
| departments now teach PhD methodology sequences and formal theory     |
| courses, though perhaps not yet with the regularity or similarity of  |
| graduate level micro, macro, and econometric training in Economics    |
| PhD programs. NSF Political Science is to be commended for its        |
| origination and support of EITM workshops and training institutes     |
| serving to link theory and methods. We urge the Foundation to find    |
| resources to permit Political Science to more actively support these  |
| human capital endeavors.                                              |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| **B.4** **[OUTCOME GOAL for ORGANIZATIONAL EXCELLENCE]{.underline}:** |
| **Providing** **"an agile, innovative organization that fulfills its  |
| mission through leadership in state-of-the-art business practices."** |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| Please see our comments above. The Political Science Program has      |
| adopted a number of innovations and is to be commended for its        |
| flexibility.                                                          |
|                                                                       |
| The COV believes that the leadership of Frank Scioli and Jim Granato  |
| has been exceptionally good specifically in three particular areas:   |
| processing grants (e.g., administration and management of the         |
| program), in new initiatives (TESS, EITM, infrastructure), and in     |
| dealing with substantial challenges (NES recompetition, budgetary     |
| scarcity, declining response rates among reviewers). We strongly feel |
| that the political science community is fortunate to have             |
| forward-looking administrators.                                       |
+-----------------------------------------------------------------------+

**2004 NSF COV Report: Sociology Program**

John D. McCarthy, Chair

Toby L. Parcel

Jan E. Stets

Milton Vickerman

# Introduction

Sociological researchers study the social causes and consequences of
human behavior, as well as social structural arrangements and social
change. The sociologist's subject matter ranges from the study of whole
societies, to large-scale institutions and organizations, to social
movements, to small groups, to individuals and their interactions. Areas
of investigation include the study of self and identities in society,
family interaction, criminal behavior, worker activity, and
organizational arrangements. Sociologists are particularly attentive to
the divisions by race, gender and social class in societies; the shared
beliefs within societies (culture); and the organization of the polity
and economy.

Sociologists develop theories about social phenomena, test these
theories through quantitative and qualitative methods, and apply their
findings to everyday life. Thus, sociologists can inform the public
about such concerns as domestic violence, company turnover, welfare
reform, divorce, and problems of war.

The NSF Sociology Program continues to be a unique and vital source of
financial support for basic research in Sociology. The demand for this
support has been rapidly increasing in recent years as seen in the
increased volume of research proposals submitted to the Sociology
Program. During the same period the budget of the Program has not
increased. Consequently more and more very strong research projects
proposed by highly accomplished researchers with strong previous records
of research productivity have gone without support.

Many of the recommendations that are included in our report are
motivated by our serious concern with this state of affairs and the
difficult dilemmas that it has created for the Program, Cluster and
Division Directors who work to increase to the appropriate level of
support for the fundamental national need of basic research in
Sociology.

##### 

##### 

##### 

##### April 1, 2004 

##### FY 2004 REPORT FOR THE SOCIOLOGY PROGRAM

#####  NSF COMMITTEES OF VISITORS (COVs)

+-----------------------------------------------------------------------+
| ##### Date of COV : March 18-20, 2004                                 |
+-----------------------------------------------------------------------+
| **Program/Cluster: Sociology/Social and Political Sciences**          |
+-----------------------------------------------------------------------+
| **Division: SES**                                                     |
+-----------------------------------------------------------------------+
| **Directorate: Social, Behavioral and Economics Directorate**         |
+-----------------------------------------------------------------------+
| **Number of actions reviewed by COV**[^2]**: Awards: Declinations:    |
| Other:**                                                              |
+-----------------------------------------------------------------------+
| **Total number of actions within Program/Cluster/Division during      |
| period being reviewed by COV**[^3]**: Awards: Declinations: Other:**  |
+-----------------------------------------------------------------------+
| **Manner in which reviewed actions were selected: Random sample**     |
+-----------------------------------------------------------------------+

PART A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures. **Provide comments in the space below the
question**. **Discuss areas of concern in the space provided.**

+-----------------------------------------------------------+----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES**  | **YES,   |
|                                                           | NO,**    |
|                                                           |          |
|                                                           | **DATA   |
|                                                           | NOT      |
|                                                           | AV       |
|                                                           | AILABLE, |
|                                                           | or NOT   |
|                                                           | APPL     |
|                                                           | ICABLE** |
+-----------------------------------------------------------+----------+
| Is the review mechanism appropriate? (panels, ad hoc      | Yes      |
| reviews, site visits)                                     |          |
+-----------------------------------------------------------+----------+
| Is the review process efficient and effective?            | Yes      |
+-----------------------------------------------------------+----------+
| Are reviews consistent with priorities and criteria       | Yes      |
| stated in the program's solicitations, announcements, and |          |
| guidelines?                                               |          |
+-----------------------------------------------------------+----------+
| Do the individual reviews (either mail or panel) provide  | Yes      |
| sufficient information for the principal investigator(s)  |          |
| to understand the basis for the reviewer's                |          |
| recommendation?                                           |          |
+-----------------------------------------------------------+----------+
| Do the panel summaries provide sufficient information for | Yes      |
| the principal investigator(s) to understand the basis for |          |
| the panel recommendation?                                 |          |
+-----------------------------------------------------------+----------+
| Is the documentation for recommendations complete, and    | Yes      |
| does the program officer provide sufficient information   |          |
| and justification for her/his recommendation?             |          |
+-----------------------------------------------------------+----------+
| Is the time to decision appropriate?                      | Yes      |
+-----------------------------------------------------------+----------+
| Comments: Across the proposals that we studied, the       |          |
| information that is sent back to the PI is fully adequate |          |
| and helpful in suggesting areas of strength in the        |          |
| proposal as well as areas for improvement. This same      |          |
| model is followed for the dissertation proposals. The     |          |
| Program Officers provide thoughtful, detailed, and        |          |
| specific justifications for recommendations. Dwell time   |          |
| seems to be relatively short, especially in view of the   |          |
| increased number of proposals that have been reviewed in  |          |
| recent rounds. The time it has taken for reviewers to     |          |
| return reviews has significantly decreased, which is an   |          |
| encouraging sign. We hope this record can be maintained   |          |
| in future rounds of proposal evaluation.                  |          |
+-----------------------------------------------------------+----------+
| Discuss issues identified by the COV concerning the       |          |
| quality and effectiveness of the program's use of merit   |          |
| review procedures: In the three-year interval for which   |          |
| data were provided, we noticed a drop in the percentage   |          |
| of ad hoc reviewers returning their reviews from over 50  |          |
| percent in 2001 to just over 30 percent in 2003. If this  |          |
| is a trend, it is somewhat worrisome.                     |          |
+-----------------------------------------------------------+----------+
| ***Recommendations: We recommend that the Program         |          |
| continue to monitor the percentage of reviewers who       |          |
| return their reviews in order to evaluate whether the     |          |
| decline in the return rate is indeed a trend. If it is a  |          |
| trend, Program Officers will want to find appropriate     |          |
| mechanisms to increase the rate of return. This could     |          |
| include prompting reviewers more frequently to increase   |          |
| the return rate for reviews.***                           |          |
+-----------------------------------------------------------+----------+

**A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers.** Provide comments in the space below the question.
Discuss issues or concerns in the space provided.

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       | Yes       |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     | Yes       |
| proposal contributes to both merit review criteria?      |           |
+----------------------------------------------------------+-----------+
| Have the *review analyses* (Form 7s) addressed whether   | Yes       |
| the proposal contributes to both merit review criteria?  |           |
+----------------------------------------------------------+-----------+
| Comments: Intellectual merit was very well addressed in  |           |
| all reviews read by COV members, in all panel summaries, |           |
| and in all Form 7s that we reviewed. The broader impacts |           |
| of the proposals were treated in a substantial number of |           |
| reviews, and virtually all panel summaries and Form 7s   |           |
| addressed broader impacts. Previous concerns expressed   |           |
| verbally to the COV in the Division about the lack of    |           |
| attention to the broader impact criterion by reviewers   |           |
| appear to have been addressed in the review process.     |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system. None.         |           |
+----------------------------------------------------------+-----------+
| Recommendations: None.                                   |           |
+----------------------------------------------------------+-----------+

A.3 Questions concerning the selection of reviewers. **Provide comments
in the space below the question. Discuss areas of concern in the space
provided.**

+---------------------------------------------------------+------------+
| Selection of Reviewers                                  | YES , NO,  |
|                                                         |            |
|                                                         | DATA NOT   |
|                                                         | AVAILABLE, |
|                                                         |            |
|                                                         | or NOT     |
|                                                         | APPLICABLE |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       | **Yes**    |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| *See program statistics for help in addressing this     |            |
| question.*                                              |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            | **Yes**    |
| appropriate expertise and/or qualifications?            |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    | **Yes**    |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| *See program statistics for help in addressing          |            |
| geographic and institutional balance; information on    |            |
| underrepresented groups is not available.*              |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      | **Yes**    |
| interest when appropriate?                              |            |
+---------------------------------------------------------+------------+
| Comments: The program officers have been successful in  |            |
| soliciting an appropriate number of reviewers for       |            |
| proposals, with a suitable mix of expertise and         |            |
| qualifications. It would be helpful to know the         |            |
| race/ethnic identification of the proposal reviewers in |            |
| order to assess whether the reviewer pool reflects a    |            |
| balance among underrepresented groups. Although these   |            |
| data are not currently available, we hope that such     |            |
| data can be produced in the future.                     |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers. None.                           |            |
+---------------------------------------------------------+------------+
| ***Recommendation: We recommend that the race/ethnic    |            |
| identification of the reviewers be produced in the      |            |
| future. This will allow a greater ability to evaluate   |            |
| whether reviewers are coming from appropriately diverse |            |
| pools.***                                               |            |
+---------------------------------------------------------+------------+

A.4 Questions concerning the resulting portfolio of awards under
review**. Provide comments in the space below the question. Discuss
areas of concern in the space provided.**

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     | Appropriate   |
| projects supported by the program.                   |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  | Not           |
| scope of the projects?                               | Appropriate   |
|                                                      |               |
| The inadequate size of grants and the issue of grant |               |
| duration is discussed below in the section on        |               |
| Program Management.                                  |               |
|                                                      |               |
| *See program statistics for help in addressing this  |               |
| question.*                                           |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   High Risk Proposals?                             |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Multidisciplinary Proposals?                     |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Innovative Proposals?                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Funding for centers, groups and awards to        |               |
|     individuals?                                     |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Awards to new investigators?                     |               |
|                                                      |               |
| *See program statistics for help in addressing this  |               |
| question.*                                           |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Geographical distribution of Principal           |               |
|     Investigators?                                   |               |
|                                                      |               |
| *See program statistics for help in addressing this  |               |
| question.*                                           |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Institutional types?                             |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Projects that integrate research and education?  |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance:                                             |               |
|                                                      |               |
| -   Across disciplines and subdisciplines of the     |               |
|     activity and of emerging opportunities?          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          | Appropriate   |
| participation of underrepresented groups?            |               |
|                                                      |               |
| *See program statistics for help in addressing this  |               |
| question.*                                           |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      | Appropriate   |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
+------------------------------------------------------+---------------+
| Comments:                                            |               |
|                                                      |               |
| [Dissertation Grant Program]{.underline}: The        |               |
| Sociology Program has supported a vigorous           |               |
| Dissertation Grant program that has resulted in an   |               |
| increased number of Dissertation grant proposals in  |               |
| 2002 and 2003. The Sociology Program has become      |               |
| known for the vitality of this grant making. The     |               |
| Program has supported approximately 45 Dissertation  |               |
| Grants each year in the 2001 to 2003 period. This    |               |
| portfolio of grants includes wide substantive and    |               |
| methodological diversity. Projects are funded in the |               |
| areas of demography, family, collective behavior and |               |
| social movements,                                    |               |
| comparative/cross-cultural/international, social     |               |
| psychology, criminology, gender, and stratification  |               |
| and inequality. Most projects include primary data   |               |
| collection.                                          |               |
|                                                      |               |
| We applaud the Sociology Program's Dissertation      |               |
| grant activities. Its successful management,         |               |
| however, has required a major investment of program  |               |
| staff time and resources.                            |               |
|                                                      |               |
| Given the increased activity of the Dissertation     |               |
| Grant Program, the COV thought it was time to        |               |
| investigate the longer-term consequences of this     |               |
| program of support for the development of the        |               |
| research capabilities of young investigators in the  |               |
| field of sociology. Do Dissertation grantees         |               |
| demonstrate a strong track record of subsequent NSF  |               |
| grant activity and funding? Do Dissertation grantees |               |
| demonstrate a strong record of subsequent research   |               |
| publication? We believe that answers to these        |               |
| questions will allow a systematic assessment of the  |               |
| long-term accomplishments of the Sociology           |               |
| Dissertation Grant Program.                          |               |
|                                                      |               |
| ***Recommendation: The Sociology Program should seek |               |
| an appropriate mechanism for reducing the amount of  |               |
| staff time and Program resources devoted to the      |               |
| review of Dissertation grant proposals. The          |               |
| Sociology Program also should find an appropriate    |               |
| mechanism to systematically track the subsequent NSF |               |
| applications and funding records as well as the      |               |
| research productivity of Dissertation Grantees.***   |               |
|                                                      |               |
| [SGER]{.underline} Grants: The Sociology Program has |               |
| used SGER grants appropriately to support higher     |               |
| risk research and initiatives. These include grants  |               |
| aimed at increasing minority representation as well  |               |
| as social issues of great current national concern.  |               |
|                                                      |               |
| The 2000 COV report encouraged the Sociology Program |               |
| to use SGER grants and dissertation grants to        |               |
| increase the representation of underrepresented      |               |
| group members. The program has used SGER grants to   |               |
| accomplish this recommendation, having supported a   |               |
| Workshop at Spelman College on Fostering Competitive |               |
| Social Science Research of Historically Black        |               |
| Colleges and Universities as well as several other   |               |
| innovative projects by minority scholars.            |               |
|                                                      |               |
| The Program responded creatively to the 9/11         |               |
| national tragedy with a series of SGER grants        |               |
| including:                                           |               |
|                                                      |               |
| Bearman, Peter/Columbia (0140024) SGER: Narrative    |               |
| networks: the World Trade Center Tragedy.            |               |
|                                                      |               |
| Bozorgmehr, Mehdi/ (01470271) SGER: How Support      |               |
| Organizations respond to Crises: Middle Eastern and  |               |
| South Asian American Organizations in the Aftermath  |               |
| of September 11.                                     |               |
+------------------------------------------------------+---------------+
| Jenkins, J. Craig/Ohio State. (01400345) SGER:       |               |
| Conflict in Central and South Asia: Improving and    |               |
| Analyzing Data from Event Reporting.                 |               |
|                                                      |               |
| Moadell, Monsoor/Eastern Michigan                    |               |
| University(0139908) SGER: A Post-Crisis Analysis of  |               |
| Attitudes and Value Orientations of the Islamic      |               |
| Publics in Egypt, Iran and Morocco.                  |               |
|                                                      |               |
| Raskinski, Kenneth/NORC (0139964) SGER: Public       |               |
| Response to a National Tragedy.                      |               |
|                                                      |               |
| A number of these grants also were supported by the  |               |
| Office of the NSF Director, as well. The portfolio   |               |
| demonstrates the value of the SGER mechanism for     |               |
| supporting high-risk research that addresses         |               |
| important national concerns.                         |               |
|                                                      |               |
| ***Recommendation: The Sociology Program should      |               |
| continue to use the SGER grant mechanism to support  |               |
| high-risk research with the potential for addressing |               |
| important national problems.***                      |               |
|                                                      |               |
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio:                                           |               |
|                                                      |               |
| The data on grants during the 2001 to 2003 period    |               |
| that were provided to COV show that female PIs       |               |
| receive lower average dollar awards than do male     |               |
| PIs. The reasons for this pattern are unclear based  |               |
| upon the available data. This pattern could occur    |               |
| because male grant recipients are on average more    |               |
| senior and thus conceptualize more costly projects,  |               |
| or it could occur because men request larger grants. |               |
|                                                      |               |
| Another concern focuses on funding the GSS. One of   |               |
| the strengths of the Sociology Program is its        |               |
| significant investment in infrastructure projects.   |               |
| These projects are important because they allow      |               |
| multiple investigators over time to test a variety   |               |
| of hypotheses using data that are derived from       |               |
| infrastructure investments. It is particularly       |               |
| heartening to see that investments are being made in |               |
| infrastructure projects beyond the GSS and the PSID. |               |
| At the same time, we have observed that the          |               |
| Sociology Program continues to spend 20% of its      |               |
| budget in support of the GSS. The quality of the GSS |               |
| is not in question, nor is its significant and       |               |
| positive impact on several disciplines. However, we  |               |
| are seriously concerned that spending 20% of the     |               |
| Sociology budget on this one project is foreclosing  |               |
| opportunities for the Program to fund innovative     |               |
| research on the cutting edges of various subfields   |               |
| in sociology and limiting the ability of the program |               |
| to fund other infrastructure work as well as         |               |
| multidisciplinary projects. Specifically, the        |               |
| program may be under funding studies of sociological |               |
| methodologies, analyses at the organizational level, |               |
| time series studies of events, broader social        |               |
| psychological processes (other than small groups     |               |
| work), and longitudinal studies of individuals. We   |               |
| note that previous COV reports have also raised the  |               |
| issue of forgone research opportunities affected by  |               |
| the high costs of funding the GSS.                   |               |
|                                                      |               |
| Another concern involves research methodology. Over  |               |
| the last few years, the Sociology Program has made   |               |
| some progress in funding a larger number of regular  |               |
| proposals using qualitative methods. These proposals |               |
| come from a variety of substantive areas within the  |               |
| field. We note, however, a relative lack of          |               |
| proposals addressing what might be identified as     |               |
| purely methodological issues, including quantitative |               |
| methodologies***.***                                 |               |
+------------------------------------------------------+---------------+
| Recommendations:                                     |               |
|                                                      |               |
| ***We recommend continued monitoring of the gender   |               |
| differential in size of grant award and further      |               |
| discussion of this issue at the next COV review.***  |               |
|                                                      |               |
| ***We recommend that the Program work with the       |               |
| Foundation to use any and all reasonable means so    |               |
| that over time the GSS is absorbing 10% or less of   |               |
| the Sociology Program's budget instead of 20%.***    |               |
|                                                      |               |
| ***We recommend that the program announcement be     |               |
| revised to explicitly invite proposals to advance    |               |
| various sociological methodologies.*** Such a change |               |
| should continue to encourage the submission of       |               |
| proposals using various methodologies, while also    |               |
| signaling that the Program is open to supporting     |               |
| work that will help to advance the study of methods. |               |
| In addition, we recommend that the program           |               |
| announcement explicitly indicate that the Program    |               |
| supports both primary data production and secondary  |               |
| data analyses.                                       |               |
|                                                      |               |
| Other recommendations include the following. A       |               |
| significant recent development in the social         |               |
| sciences has been the construction of large,         |               |
| multidisciplinary projects to address major issues   |               |
| that are informed by several disciplines. Sara       |               |
| McLanaghan's project on Fragile Families is a case   |               |
| in point, where funding is derived from several      |               |
| governmental and foundation sources, and where       |               |
| investigators from more than one discipline combine  |               |
| their expertise on both quantitative and qualitative |               |
| methods in pursuit of key questions***. We recommend |               |
| that the Sociology Program contribute to the funding |               |
| of several of these large projects as a vehicle for  |               |
| ensuring that some program funds are being devoted   |               |
| to these most significant endeavors.***              |               |
|                                                      |               |
| Additionally, over the last few years, the program   |               |
| has held workshops to bring together scholars        |               |
| concerned with fundamental issues in the discipline  |               |
| and in related disciplines. Examples include the     |               |
| workshop on understanding global tensions, on        |               |
| repeated cross-sectional and longitudinal surveys,   |               |
| and on the scientific foundations of qualitative     |               |
| research. These are very important activities for    |               |
| creating national and international conversations on |               |
| key issues, and on sharing expertise on vital        |               |
| topics. ***We recommend that the Sociology program   |               |
| continue to sponsor such activities as a vehicle for |               |
| promoting quality research, both disciplinary and    |               |
| interdisciplinary, and encouraging networking around |               |
| these most vital topics. We recommend that Program   |               |
| Officers continue to devote significant time to this |               |
| area of the program portfolio.***                    |               |
|                                                      |               |
| ***We also recommend that the Sociology Program      |               |
| develop new initiatives within the Cluster and       |               |
| Division that will address cutting edge developments |               |
| in the field, and in related fields, and seek to     |               |
| obtain new funding for these activities.*** The      |               |
| example of the EITM initiative developed by          |               |
| Political Science provides one model that should be  |               |
| studied as a "best practice." Although Sociology did |               |
| benefit from this initiative, we recommend that the  |               |
| Sociology Program take the lead in comparable        |               |
| innovation in the future.                            |               |
|                                                      |               |
| Finally, we applaud the efforts of the Sociology     |               |
| Program Officers in their extensive efforts to       |               |
| favorably position the Program in the new HSD        |               |
| initiative. ***In addition, we recommend that the    |               |
| Sociology Program continue to devote sufficient time |               |
| and effort to assuring that the Program is well      |               |
| positioned for new NSF wide initiatives.***          |               |
+------------------------------------------------------+---------------+
| **A.5 Management of the program under review.**      |               |
| Please comment on:                                   |               |
|                                                      |               |
| The Committee discussed at length three important    |               |
| issues related to the management of the Sociology    |               |
| Program. These include the management of budget      |               |
| constraints, the history of rapid turnover of the    |               |
| Program Rotator, and the workload of the Program     |               |
| staff that results from the increased rate of        |               |
| proposal submission combined with effort devoted to  |               |
| other extra-program tasks, especially new NSF        |               |
| initiatives.                                         |               |
|                                                      |               |
| Over the 2001-2003 period, the budget of the         |               |
| Sociology program remained flat, while there was a   |               |
| corresponding 66% increase in proposals submitted to |               |
| the program. The increase in submissions is          |               |
| commendable. It suggests that the Sociology Program  |               |
| leadership is being successful in outreach. Given    |               |
| the increase in submissions, the Program Officers    |               |
| have worked hard to maintain a 25% funding rate for  |               |
| regular research projects. This rate continues to    |               |
| serve to encourage investigators to submit their     |               |
| proposals. In an effort to offset the many forgone   |               |
| research opportunities associated with a flat budget |               |
| and increasing submission rate, the Program Officers |               |
| have been diligent in seeking funds from other       |               |
| programs through joint reviewing of proposals and    |               |
| new initiatives. These efforts have significantly    |               |
| helped support work that ordinarily would not have   |               |
| been supported from the core funding of the          |               |
| Sociology Program. However, there remain a number of |               |
| concerns about the budget.                           |               |
|                                                      |               |
| In order to cut costs and still fund good            |               |
| sociological work, the Program has been creative in  |               |
| stretching its dollars. For example, the Program has |               |
| streamlined exactly what costs are eligible to       |               |
| funding for any one project. For example, the        |               |
| Program does not allow academic year release time    |               |
| for PIs in Research I institutions, PI summer salary |               |
| is capped at \$15,000 per year, and no travel to     |               |
| professional meetings is allowed. This streamlining  |               |
| necessarily puts added burdens on PIs and could      |               |
| compromise the accomplishment of significant         |               |
| research, as well as the dissemination of research   |               |
| results. Perhaps more importantly, we find that over |               |
| the three-year period, awards have become shorter in |               |
| duration, and the average size of budgets has become |               |
| smaller. Again, while this allows the program's      |               |
| dollars to be stretched further, it makes it more    |               |
| difficult for PI's to carry out their sociological   |               |
| research. It may limit what can be studied both      |               |
| theoretically and methodologically.                  |               |
|                                                      |               |
| Over the last five or six years, there has been a    |               |
| tendency in the Sociology Program for quick turnover |               |
| of rotators. Rotators have tended to serve only one  |               |
| or two years at the longest. The Program should make |               |
| every effort to encourage rotators to stay for       |               |
| longer periods of service\-- 3 to 4 years rather     |               |
| than only 1 to 2 years, assuming good performance by |               |
| the rotators. Extending the terms of rotators would  |               |
| lend more continuity to the program while at the     |               |
| same time reducing the Permanent Program Officer's   |               |
| effort devoted to the role of "trainer."             |               |
|                                                      |               |
| The workload of the Sociology Program has doubled    |               |
| over the last several years as the result of         |               |
| increased proposal submission (both regular grant    |               |
| and dissertation), participation in initiatives, and |               |
| the Program Officer's duties as Cluster Manager. In  |               |
| spite of this increased workload, program leadership |               |
| has been characterized by innovation. We commend     |               |
| this innovative leadership such as the development   |               |
| of good workshops, seeking funding from other        |               |
| programs for support of proposals, and participation |               |
| in larger NSF initiatives such as Human and Social   |               |
| Dynamics. It is absolutely critical that there be    |               |
| continued and strengthened innovative leadership in  |               |
| order to enhance and develop the program's           |               |
| capabilities to support the most important work both |               |
| disciplinary and interdisciplinary, especially that  |               |
| which informs broader initiatives such as HSD.       |               |
+------------------------------------------------------+---------------+
| Responsiveness of the program to emerging research   |               |
| and education trends.                                |               |
+------------------------------------------------------+---------------+
| Program planning and prioritization process          |               |
| (internal and external) that guided the development  |               |
| of the portfolio under review.                       |               |
+------------------------------------------------------+---------------+
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the management of the program.                       |               |
+------------------------------------------------------+---------------+
| Recommendations:                                     |               |
|                                                      |               |
| Given the above discussion on the management of the  |               |
| program, we make several recommendations. ***First,  |               |
| we strongly urge NSF to allocate more funds to the   |               |
| core of the Sociology Program.*** Given future       |               |
| prospects of the submission rate continuing to       |               |
| remain high, the Committee views the budgetary       |               |
| picture as quite worrisome.                          |               |
|                                                      |               |
| ***Second, the Sociology Program, the Cluster        |               |
| Administrator as well as the Division Administration |               |
| should explore mechanisms that would serve to extend |               |
| the average tenure of Sociology Program Rotators.*** |               |
| Longer terms for rotators would allow the Program    |               |
| Officers to devote more time to innovative and       |               |
| strong leadership. The recently increasing workload  |               |
| may be an inhibiting factor for convincing           |               |
| candidates to remain in the rotator post for an      |               |
| extended time. Incentives for rotators to stay       |               |
| longer than the minimum 1 to 2 years should be       |               |
| investigated. One possibility may be to find ways in |               |
| which rotators can continue to carry out their own   |               |
| research during their service to NSF.                |               |
|                                                      |               |
| ***Third, the needs of additional staffing for the   |               |
| Sociology Program and the Cluster should be given    |               |
| the highest priority by NSF.*** Staff personnel      |               |
| should be increased to offset the increased workload |               |
| of the Program Officers. A very important            |               |
| consequence of increasing staff personnel is that it |               |
| will enable Program Officers to have more time to be |               |
| proactive and plan new research initiatives that     |               |
| could be put forward when budgets are being          |               |
| considered. Adding an additional Program Officer     |               |
| within the cluster could help address some of the    |               |
| workload issues we have identified.                  |               |
|                                                      |               |
| Finally, as a result of the efforts of Sociology     |               |
| Program Administrators to accomplish appropriate     |               |
| outreach activities aimed at developing responses by |               |
| sociological researchers to new NSF initiatives, the |               |
| need for them to travel to professional meetings as  |               |
| well as other suitable venues has become             |               |
| increasingly important, and facilitating their       |               |
| travel is imperative. The ability of the Program     |               |
| Directors to travel, however, is constrained by      |               |
| budgetary considerations. ***The budget allocated to |               |
| travel related expenses for Sociology Program        |               |
| Officers should be supplemented to facilitate the    |               |
| appropriate level of outreach activities.***         |               |
+------------------------------------------------------+---------------+

B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.

+-----------------------------------------------------------------------+
| **B.1 [OUTCOME GOAL for PEOPLE]{.underline}: Developing "a diverse,   |
| competitive and globally engaged workforce of scientists, engineers,  |
| technologists and well-prepared citizens."**                          |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| NSF grants have contributed to the development of "a diverse,         |
| competitive and globally engaged workforce of scientists, engineers,  |
| technologists and well-prepared citizens." Over the past three years  |
| women have received approximately fifty percent of regular NSF grants |
| and 9 percent of grants have gone to minority applicants. These       |
| grants have supported high quality research.                          |
|                                                                       |
| For instance, Mansoor Moaddel's, "In Search of A Sociopolitical       |
| Community: The Cases of Egypt, Iran and Jordan" (#0097282: Eastern    |
| Michigan University) has demonstrated notable value differences among |
| Islamic societies. Iranians are both less religious and more          |
| nationalistic than either Egyptians or Jordanians; they display more  |
| liberal attitudes toward women and work outside the home; and, in     |
| general, Muslim women in these countries are less religious than men. |
|                                                                       |
| Robert Smith's, "Determinants of Education and Mobility among         |
| Second-Generation Mexican-Americans" (#9731280: Barnard College) has  |
| shown that the assimilation of Mexicans in the United States is       |
| shaped by such factors as gender, the proportion of adolescents in    |
| the immigrant population, and their relationship with African         |
| Americans.                                                            |
|                                                                       |
| Theda Skocpol's, "The Political Incorporation of Immigrants:          |
| Citizenship and Participation in the United States and Canada"        |
| (#l0000310: Harvard University) has investigated the reasons for      |
| differential rates of citizenship and political participation between |
| Canada and the United States. The study concludes that this variation |
| stems from the level of government support for immigrant political    |
| participation. The more active this support, the more likely it is    |
| that immigrants will nationalize and participate in the political     |
| process.                                                              |
|                                                                       |
| These studies demonstrate the cross-cultural, theoretical, and        |
| practical nature of the work that is being conducted by scholars who  |
| have received NSF grants. Moreover, the majority of regular grants    |
| help fund graduate students and, sometimes, undergraduates.           |
| Consequently, NSF grants are helping to train a new generation of     |
| scholars in Sociology.                                                |
+-----------------------------------------------------------------------+
| **B.2 [OUTCOME GOAL for IDEAS]{.underline}: Enabling "discovery       |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The portfolio of NSF grants in the Sociology program include many     |
| projects that enable discovery across the frontier of science,        |
| innovation and service to society. The mix of grants is especially    |
| strong on projects that bring innovative ideas and methods to bear    |
| upon problems of wide societal relevance.                             |
|                                                                       |
| The following projects are among a large number of exemplary research |
| products that illustrate the consequence of NSF support for           |
| generating research incorporating exciting new ideas.                 |
|                                                                       |
| Uggen, Christopher/Minnesota(9819015) The Political Consequences of   |
| Felon Disenfranchisement in the United States. This groundbreaking    |
| project demonstrates that millions of current and ex-felons are       |
| legally disenfranchised, that great differences exist between states  |
| in the conditions of disenfranchisement and the growth of the size of |
| this population of citizens unable to vote has had direct             |
| consequences upon electoral outcomes in recent decades.               |
|                                                                       |
| Lichter, Daniel/Ohio State. (0096153) Entering Adulthood: Poor,       |
| Socially Disadvantaged and Resilient. Daniel Lichter's work provides  |
| evidence that increasing the marriage rates for low-income families   |
| would reduce their reliance upon public assistance and promote the    |
| well-being of children.                                               |
|                                                                       |
| Jenkins, Craig/Ohio State. (9710958) Mass Political Conflict: Origins |
| and Impact on Political Change. This timely project creates a         |
| conflict carrying capacity index that can serve as an early warning   |
| tool for assessing trends toward political destabilization in a       |
| nation that portends mass political conflict. Jenkins shows that the  |
| major risk factors for internal wars are autocratic government,       |
| poverty, and forced ethnic political incorporation.                   |
|                                                                       |
| Menken, Jane/University of Colorado. (9910662) Female Work, Public    |
| Policy and Fertility in Developed Nations. This project by National   |
| Academy of Sciences member Jane Menken shows, among other important   |
| findings, that public policies relating to tax exemptions for         |
| dependents and child care credits influence both age-specific         |
| fertility rates as well as the supply of women workers. Public        |
| financial incentives can alter the costs of childbearing and,         |
| thereby, affect aggregate fertility rates.                            |
|                                                                       |
| Pager, Devah/Northwestern (0101236) (0318158) Race, Jobs and Prison:  |
| The Consequence of Incarceration for Racial Stratification/           |
| Discrimination in Low Wage Labor Markets: An Audit Study for New York |
| City. This project investigates the impact of incarceration upon      |
| employment prospects using an experimental design. Initial results,   |
| that are presently being tested more rigorously, show that all other  |
| things being equal, employers are less likely to call back potential  |
| employees after an interview if they indicate they have been          |
| incarcerated, and that black applicants are less likely to be called  |
| back than are white applicants. Pager received a Dissertation grant   |
| for the early phases of this project, and has a current grant for a   |
| more comprehensive project.                                           |
+-----------------------------------------------------------------------+

Dobbin, Frank/Princeton (0004333) Affirmative Action at Work: Corporate
Compliance Activities and Workforce Composition. Publicly available
annual reports of hundreds of large U.S. employers are used, along with
surveys of the company's employment practices to make the first
large-scale sustained effort to determine the effects of employment
practices on workforce composition and to determine the effectiveness of
activities designed to satisfy anti-discrimination laws in making the
workforce more diverse.

Granovetter, Mark/Stanford University (9710531) Reciprocity in the
Russian Labor Market: Its Role in the Transition from State Socialism.
This innovative project has studied the emerging labor market in Samara,
Russia during the transition from state socialism. Preliminary results
of the project show that personal networks of family and friends are
crucial in explaining the decisions of employers about whom to hire, but
that large numbers of those hired were not actively seeking work. Men,
when they find work, are paid twice as much as women and women find
fewer suitable jobs when they seek them.

Cathyrn Johnson/Emory University (9730445) Analyzing the Effects of
Formal Position and Leader Legitimacy on Leader-Subordinate Interaction.
This experimental study explores how a leader's legitimacy shapes
subordinates' responses to and perceptions of a leader's competence and
power. The research has direct implications for how formal authority and
leader legitimacy influences the effectiveness of leaders.

Cecilia Ridgeway/Stanford University (0000201) The Gendered Selection of
Activities and the Reproduction of Gender Segregation in the Labor
Force. This project combines experimental and survey methods to show how
cultural beliefs about gender channel men and women into different
career paths, and thereby severely constrain career choice processes.
These choices, then, indirectly contribute to the perpetuation of gender
segregation in the labor force.

+-----------------------------------------------------------------------+
| **B.3 [OUTCOME GOAL for TOOLS:]{.underline} Providing "broadly        |
| accessible, state-of-the-art S&E facilities, tools and other          |
| infrastructure that enable discovery, learning and innovation."**     |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The Sociology Program is being very successful in developing tools    |
| and infrastructure to benefit several disciplines. The GSS is an      |
| extremely successful project that benefits not only sociology, but    |
| also a variety of other disciplines, some of which are not directly   |
| supported by NSF. Indeed, 42% of the publications that are derived    |
| from the GSS are produced by non-sociologists. The project has been   |
| ongoing for so long and has produced such a volume of output that it  |
| is truly a significant part of infrastructure in the social sciences. |
| The Sociology program also provides support to the PSID, which is     |
| another significant part of infrastructure in the social sciences.    |
| The PSID is often used by economists, although a variety of scholars, |
| including sociologists, use this data base to study the economic      |
| well-being of families and individuals over time.                     |
|                                                                       |
| Steve Ruggles' (0110788, Minnesota) work with the North Atlantic      |
| Population Project will construct a compatible integrated data base   |
| of micro census data from the late 19^th^ century in Canada, Great    |
| Britain, Iceland, Norway and the United States. This project will     |
| allow investigators in demography, stratification, historical         |
| sociology, family and other fields the ability to do significant      |
| comparative research on a variety of topics on individuals and        |
| households.                                                           |
|                                                                       |
| Kathryn Edin (0127123, Northwestern) is adding important qualitative  |
| and longitudinal data to the social science infrastructure. She is    |
| completing a third wave of interviewing with 75 low- to               |
| moderate-income people, mostly unmarried, who had a child in 2000 in  |
| Chicago, Milwaukee or New York. The data will provide accounts of     |
| couple dynamics, and parent-child relationships, as well as various   |
| forms of material and social support provided to children.            |
|                                                                       |
| Mansoor Moaddel (0097282, 0242862, Eastern Michigan University) is    |
| conducting surveys of nationally representative samples of 1,000      |
| citizens in Egypt, Iran and Jordan using face-to-face interviews, and |
| of 1,500 Saudis. The data will be used by a variety of investigators  |
| to study similarities and differences in values across these          |
| societies, with particular attention to the politics, religion,       |
| community, the place of women in society, Islamic fundamentalism, and |
| Western cultural invasion.                                            |
|                                                                       |
| Randy Hodson (0112434, Ohio State) has created an organizational      |
| ethnographic data base by content analyzing case studies of           |
| organizations published in monographs. Systematic coding of the       |
| ethnographies has resulted in a large data base that will enable      |
| researchers to study topics such as job stress, worker solidarity,    |
| the role of information technology and race and gender in the         |
| workplace across a diverse sample of 142 organizations.               |
|                                                                       |
| **B.4 [OUTCOME GOAL for ORGANIZATIONAL EXCELLENCE]{.underline}:       |
| Providing "an agile, innovative organization that fulfills its        |
| mission through leadership in state-of-the-art business practices."** |
|                                                                       |
| Comments: None.                                                       |
+-----------------------------------------------------------------------+

##### PART C. OTHER TOPICS

C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.

C.2 Please provide comments as appropriate on the program's performance
in meeting program-specific goals and objectives that are not covered by
the above questions.

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

> We think the Sociology program's performance would be enhanced by
> greater time being devoted to planning within the SBE Division. Such
> planning must include other cluster programs and Division
> representatives. Planning sessions should be devoted to developing
> ideas for new initiatives, talking about new developments within and
> across disciplines, and developing strategies for garnering additional
> resources to support scientific innovation in the Division.

***Recommendation: We recommend that regular planning sessions for
Program Officers and Division Directors be initiated that will be
devoted to developing new initiatives and funding strategies for the
division.***

C.4 Please provide comments on any other issues the COV feels are
relevant.

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

> The Sociology COV found the process to be a very productive experience
> and applaud the efforts of Sociology Program and Cluster and Division
> Directors in providing materials concerning the grant files as well as
> summaries of Program activities. We could not have grasped in so short
> a time the many successes as well as the areas we believe are in need
> of improvement in the Sociology Program without the help of such
> admirable staff work. In spite of our general praise for the process
> as we experienced it, however, we have some suggestions for how to
> improve the process for future COVs. These suggestions include ones
> concerning scheduling, technology and data quality.
>
> [Scheduling:]{.underline} The time available for reviewing materials,
> discussing them, identifying successes and problem areas in a Program
> during the COV process is very short. As a result, it is imperative
> that COV members be given the maximum amount of uninterrupted time to
> focus upon their assessment and report writing tasks. We found the
> disruptions that resulted from the scheduling on the first day, and
> the necessity of moving between rooms and floors for cluster wide
> meetings to severely reduce the time we had to devote to our primary
> tasks. We found the difficulties of access to the building and
> difficulties with moving between floors on the Saturday of our visit
> to also be disruptive with similar consequences for the amount of
> uninterrupted time we could devote to our primary tasks.
>
> In the future, as well, it would be advisable for the Division
> Director to schedule separate meetings with each COV Program
> Committee. We found the meeting we were able to schedule with the
> Director quite valuable, but it occurred as a result of our request
> because the originally scheduled meeting included COV members from
> each of the three cluster Programs.
>
> [Technology:]{.underline} We found the arrangements for report writing
> to be cumbersome. We agreed that it would have been very helpful if
> the laptop computers we used for writing had been networked together
> to help smooth the workflow.
>
> [Data Quality:]{.underline} Financial data about Sociology program
> core support and Program expenditures is vital to the work of the COV
> for assessing the past performance of the Program, Cluster and
> Division as well as future prospects. We discovered in our
> deliberations and in discussions with Program Directors that the
> financial trend data that was made available to us was not easily
> interpreted***.***
>
> ***Recommendation: For future COVs, we recommend that the financial
> data for the Sociology Program, associated clusters and the division
> be presented in a more transparent form, and, especially, that a
> budget narrative be attached to facilitate COV interpretation of the
> figures that are provided.***

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the Sociology Program

John D. McCarthy, Chair

[^1]: The Committee of Visitors for Political Science consisted of Frank
    R. Baumgartner, Penn State University (Chair), Kathleen McGraw, Ohio
    State University, William Mishler, University of Arizona, and
    Kenneth Shepsle, Harvard University,.

[^2]: To be provided by NSF staff.

[^3]: To be provided by NSF staff.