**Committee of Visitors Report:**

## Information and Intelligent Systems Division

# National Science Foundation

## Review Period: FY 1999 - FY 2002

July 28, 2003

**Committee Members:**

**George Bekey, Richard Belew, Jonathan Grudin, Ricardo
Gutierrez-Osuna,**

**Vicki Hanson, Thomas Huang, Judy Olson, Neil Rowe, Elliot Soloway
(Chair)**

**Section 1: Core Findings**

As stated in the Guidelines for COV, a COV primary charge is to examine
"the integrity and efficiency of processes used to solicit, review,
recommend and document proposal actions," [^1] The COV2003 Committee
finds that, indeed, the IIS proposal review process does possess the
required "integrity." And, with respect to "efficiency" COV2003 feels
that, while improvements can definitely be made, the review process is
an efficient one. (Below, we provide specific recommendations that we
feel should improve the efficiency of the IIS process.) Thus, at the
outset, COV2003 goes on record as enthusiastically endorsing the IIS
program for the reporting period, 1999-2002.

We hasten to point out that IIS achieved this success in the face of a
number of factors:

-   There was steady growth in submissions, which put significant
    pressure on the system.

-   ITR funds, while they were most welcome, again served to put
    significant pressure on the review system.

-   Several agencies who had originally committed resources to IIS
    (Digital Libraries Program) withdrew that support, leaving IIS to
    honor its commitments.

-   Reporting requirements (e.g., GPPR) increased putting still further
    demands on program directors and staff.

-   Information technologies came online (e.g., proposal review,
    Fastlane) that, while in the long-run have had a positive impact on
    work practices, nonetheless imposed real demands on IIS personnel.

-   Finally, there has been a considerable turnover in IIS program
    directors and division director.

The fact that IIS did perform as well as did it did in light of the
above points to the high quality of the IIS organization and team.

This COV focus was on IIS processes. Rather than devoting our limited
time to analyzing and predicting trends in the IIS content areas, our
goal was to recommend a set of processes that can guide IIS in evolving
its content focus. Our time at NSF was spent as follows:

-   Monday, June 9: The IIS Program Directors, Division Director, and
    CISE AD provided us with in-depth presentations along with question
    and answer periods.

-   Tuesday, June 10: We spent the entire day reviewing the jackets IIS
    provided. We requested additional jackets and additional data which
    were immediately provided to us.

-   Wednesday, June 11: We spent the morning organizing our report and
    presenting to the CISE AD and the IIS Division Director.

The organization of this document is as follows:

-   Section 2: An analysis of previous COV reports and their impact
    within IIS

-   Section 3: IIS Division: Issues and Recommendations

-   Section 4: IIS Programs: Issues and Recommendations

**Section 2: An Analysis Of Previous COV Reports And Their Impact Within
IIS**

An examination of the previous three COV reports (1992, 1995, and 1999)
shows that the IIS Division has successfully addressed many of the
long-standing issues, sometimes as part of a Foundation-wide effort.

The recruitment of qualified reviewers has been significantly improved.
Past practices, consisting of unsolicited mailings to potential
reviewers, have been replaced by the more efficient process of
contacting reviewers beforehand and inviting them to participate in the
initial assignment of proposals. Following previous COV recommendations,
IIS has continued to rely almost exclusively on review panels, as
opposed to ad-hoc mail reviews.

A significant reduction in review turn-around times, which was a
consistent recommendation of previous COV, has been achieved during the
1999-2002 period. Statistics collected by the IIS Division in
preparation for the 2003 COV show that the number of IIS proposals with
a dwell time less than 6 months has increased from 27% in 1999 to 70% in
2002.

An increased use of information technology in the review process, also a
recurring recommendation of past COV, has been implemented during the
1999-2003 period with the introduction of the electronic Panel Review
System through FastLane.

IIS has also been successful in attracting senior researchers to serve
as program directors during the period of this COV review. Drs.
Thuraisingham and Yuh have been recruited as Program Directors for the
IDM/DAS and RVC programs, respectively, and a new PD is being recruited
for the AICS program.

Nonetheless, a long-standing issue remains to be resolved: past COV have
consistently indicated that Program Directors are burdened with
significant workloads. The current COV also sees this situation and will
make recommendations towards reducing the PDs burden.

**Section 3: IIS Division: Issues and Recommendations**

**Section 3.1: Issues Impacting Review Process & Post-Award Process**

First we identify several issues that impact (some positively and some
negatively) the efficiency and effectiveness of the review process and
the post-award process.

-   **Technology-supported panel reviews**: After looking at the
    proposal jackets and drawing on the COV experiences with panel
    reviews, we felt that the technology used in the panel reviews
    provides a more effective review process. For example, the
    technology eliminates paper handling and handwritten notes,
    encourages reviewers to be more thorough in their reviews, etc.

-   **Program directors positively influence panel decisions**. Program
    directors walk a fine line during a panel review; based on our
    discussions with program directors and our perusal of the jackets,
    we felt that program directors provided valuable input during a
    panel review.

-   **Errors in the data we were presented with**: In a number of the
    reports we were given that dealt with proposals (e.g., scoring,
    rankings, number of submissions) we found missing information and
    incorrect information. Such a situation does not breed confidence
    that we -- or IIS -- is truly making data-driven decision making.

-   **Broad Impact Statement & Research and Education Plans seem to be
    boilerplate**. Frankly, in reading those sections of the proposals
    we reviewed, we felt that the proposers were simply providing vague
    platitudes.

-   **NSF-sponsored research reports are hard to access/find.** While
    technology has clearly assisted the review process, the COV hasn't
    seen where it has supported the dissemination of research findings.

-   **Lack of use of SGERs**. We found only very limited use of SGERs by
    PDs.

**Section 3.2: Recommended Changes**

In order to address the above issues, we suggest the following:

-   **Redistribute the work using information technology**: The
    technology used to support the panel reviews is an excellent example
    of how IT can significantly improve a process. Drawing on that
    experience, we suggest that IT be used at a number of points in the
    review and post-award process to capture readily-available
    information that can then be more easily disseminated. For example,
    during the review process, reviewers can indicate whether they find
    the proposal under review to be one of high risk/high gain. Still
    further, asking awardees to fill out a form, periodically, that goes
    onto the web (or at least goes into NSF's database for
    summarization) that highlights that quarter's activities is not an
    onerous chore. Indeed, it only helps to support a productive
    researcher. Thus, capturing these sorts of information ***during***
    the various processes makes it much easier, then, to compile reports
    and/or publish information on the web. The COV felt that a careful
    analysis of the reporting demands made on the PDs could in fact be
    off-loaded to reviewers and proposers without unduly burdening these
    individuals since the information being captured was already being
    generated by the reviewers and proposes. In effect, then, the COV
    recommends that IT continues to be used to automate more of the
    processes.

-   **Reflect on decision-making anomalies**. We found evidence of a
    number of funding decisions which were contrary to, or at least not
    consistent with, panel recommendations. Other proposals were moved
    across different programs, etc. We applaud the initiative and
    insight PDs display in making such decisions. But these situations
    are also opportunities for discussion and adaptation of the process.
    For example, there may have been potentially confusing language in a
    program call, poor instructions given to a panel, etc. We feel that
    public discussion of the decision and its implications should be
    documented.

-   **Increase base funding level.** We have noted in a number of
    programs that funding levels for individual grants have decreased
    and decreased over time. Oftentimes one graduate student plus one
    summer month of the PI is simply not enough resource to do a
    computer-based project. We encourage IIS to try to provide at least
    two graduate students in a research project, and in general, stop
    the decrease in base funding levels.

-   **Disseminate, disseminate, disseminate.** We can't stress enough
    how important it is to have NSF's work be more accessible. For
    potential proposers, for the public at large, for our politicians,
    for our international colleagues -- they all can benefit from
    gaining access to more of the work being produced under NSF funding.
    (For example, reports from NSF supported workshops are nearly
    impossible to locate, even when they result in new initiatives. We
    suggest posting them on an NSF server, or posting a pointer to the
    workshops' websites, so the reports become accessible to the larger
    community.) In asking for increased dissemination, we are not
    meaning to increase the workload of IIS staff. Rather, through the
    use of information technology, a great deal of the information that
    would be valuable to a range of constituencies should be capturable
    and presentable automatically.

-   **Make declinations timely.** Not only need IIS strive to make
    decisions in the six month window, but IIS needs to be sensitive to
    the academic calendar in its acceptances and declinations.

-   **Create panel review standards (ranking, feedback, managing).** In
    our review of panels across the various programs, we found
    differences in how they carried out their panel reviews (e.g., some
    programs used a 3 point ranking scale, some a 2 point scale). We
    strongly suggest that some common standards be put in place as soon
    as possible.

-   **Continue reducing overlap across programs.** As noted below, in
    the period of the COV we found proposals funded in one program that
    seemed to fit better in another. While there will always be some
    problematic situations, we encourage IIS to continue to strive to
    minimize the overlap in programs.

-   **Panelists enforce importance of "broader impact" and "research and
    education**." One strategy to encourage proposers to take seriously
    these two sections would be for the reviewers to place more weight
    on them in their reviews. If reviewers actually lowered the rankings
    of proposals for failing to provide actionable "research and
    education" plans or for mouthing vague platitudes on the "broader
    impact" section, then proposers would take those sections more
    seriously.

-   **Enable program officers to attend conferences.** Program directors
    need to attend conferences to mix with the researchers and help
    build the community. We strongly urge that travel funds be provided
    to permanent NSF staff for such travel.

-   **Leave ITR dollars in IIS**. ITR funds have enabled new types of
    research -- and most importantly, have grown the research community
    and its capacity to do new work. As ITR per se disappears, we
    strongly encourage NSF to provide to IIS funds commensurate to the
    ITR funding levels. These funds should continue to support
    large-scale work, multi-investigator, and infrastructure-type
    research projects.

-   **Annex areas that intensively employ IIS technologies.** Areas such
    as bioinformatics, educational technology need to be brought into
    the IIS fold explicitly. Areas such as these advance computing
    technology in the context of their content area.

-   **Create explicit support for software infrastructure for IIS
    communities**. CISE supports the development of software (e.g., the
    middleware initiative) that provides infrastructural software for
    CISE-area projects. IIS communities need similar infrastructural
    software support. We leave to the IIS the specific mechanics to be
    used to solicit, review, and fund proposals in this area.

-   **Continue collaborating with other programs.** IIS already works
    with other divisions, e.g., EHR, SBE. We applaud this non-stovepipe
    activity and we encourage IIS to continue working with others in
    NSF.

-   **Encourage the reporting of "negative" results.** As scientists we
    recognize that full reporting of negative results could accelerate
    progress by enabling subsequent researchers to avoid dead ends,
    anticipate methodological problems, accurately gauge requisite
    resources, and so on. Yet the culture of science and engineering
    does not reward, and may punish, failure. A grant recipient may
    report the approach that succeeded and avoid mentioning the four
    that did not, anticipating that a record of one in five will not
    look good when a new grant is sought. By taking the lead in
    encouraging the reporting of negative results, IIS could do a
    tremendous service to the field.

-   **The COV considers that it would be beneficial to build a memory
    mechanism into the review process, in a manner similar to what is
    done at NIH and journals**. Enormous effort goes into writing a good
    grant proposal, and the large majority that have their proposals
    rejected deserve a meaningful dialog, or it is hard to rationalize
    trying again. If a "memory" of earlier reviewers' criticisms and
    proposer's responses would seem to make the process much more
    efficient for all concerned, at least in some cases.

> However, as pointed out by the IIS Division Director, this would
> require foundation-wide changes. A simple memory mechanism would be
> for NSF to encourage PIs resubmitting proposals to address the
> concerns of the previous panel, perhaps in a separate cover letter. In
> encouraging, rather than enforcing, the PD clerical duties are not
> increased, and the PI is still allowed to choose a \"fresh start\".
> Building memory into the system benefits the NSF programs in two
> significant ways:

-   By explicitly addressing weaknesses of previous submissions, it can
    be expected that the overall quality of the proposals will be
    increased. Several Program Officers pointed out that they would like
    to see more quality, rather than more quantity.

-   Given the current acceptance ratios, the cost-effectiveness of
    submitting a proposal to NSF has been questioned. Resubmission would
    be a mechanism to leverage previous efforts by the PI.

**Section 3.3: Recommendations for Division Renewal, Growth and
Adaptation**

The real challenge as IIS moves forward is dealing with change, e.g.,
making changes to its content programs in a timely fashion to include
new areas of research, and managing the transitions so that the research
infrastructure and research teams move along with those changes in
funding. Based on the events of the past, the processes needed to
address that challenge must also take into consideration the external
and internal pushes and pulls that will inevitably take place, e.g.,
turn over in staff, increased reporting demands, unplanned for changes
in partner's funding. Thus, IIS needs to put in place a flexible,
explicit set of processes that will enable it to deal with change
effectively and in so doing create a mechanism for IIS's constant,
ongoing renewal, growth and adaptation.

Herewith, then, is this COV recommendation for a Four Step Process that
should enable IIS to identify new research areas that deserve attention.
The activities and mechanisms themselves that make up that process are
all already in place in IIS. However, our contribution is in seeing how,
working in concert, the following activities can serve as the basis for
ongoing, continual growth and change.

-   **Step 1 -- SGERs and Planning Grants: Giving Researchers a Voice
    for Change** -- While this COV has pointed out that SGERs are not
    uniformly used in IIS and that planning grants no longer seem to be
    awarded, we feel strongly that these funding mechanisms can serve as
    "early detection" activities for new areas of research that might
    well deserve fuller support. With the technologies in the IIS
    umbrella changing so rapidly -- indeed, the pace of that change has
    recently been increasing -- a central, top-down planning process is
    sure to miss many potentially interesting new developments. SGERs
    and Planning Grants let the ideas bubble up from the field. SGERs
    empower individual researchers to venture forth, while Planning
    Grants enable ad hoc, dynamic collaborations to form quickly in
    response to and in advance of new developments. While care must be
    exercised in balancing the funding of SGERs and Planning Grants with
    regular grants, IIS is in no danger of abusing these funding
    mechanisms.

-   **Step 2 -- Workshops: Giving the Community a Voice for Change** --
    Based on the findings from SGERs and Planning Grants, IIS can then
    fund workshops that bring researchers together to nurture the growth
    of new communities. Ideas and research are products of a individuals
    and of communities. For a relatively small amount of money, a
    workshop can held where the new and the experienced come together to
    work through new ideas, old ideas whose time has come, etc. This COV
    has seen that workshops were definitely a part of the funding
    portfolio over the past 3 years. However, the rationale for why a
    workshop was or was not funded was less clear. In contrast, we
    propose that at least one of the criteria for support/not support of
    a proposed workshop is its relationship to the findings of an SGER
    (or several SGERs) and/or a Planning Grant (or several Planning
    Grants).

-   **Step 3 -- Initiatives: Trial Support for a New Community** -- NSF
    in general and IIS in particular has used Initiatives as a mechanism
    to supercharge new research areas. The rationale for Initiatives, in
    our proposed plan, would to some extent at least, arise from the
    findings of the Workshops.

-   **Step 4 -- Programs: Longer-term Support for a Community** -- And
    finally, for those research areas where a large, vibrant community
    is growing up, IIS may well decide to create a program, and make
    that community a first-class citizen.

IIS has extensive experience in creating and folding funding mechanisms
of the last two steps. This COV urges IIS to renew its commitment to the
funding mechanisms in the first two steps in order to more rationally
identify and explore potentially valuable research areas. In embracing
the Four Step Process, IIS will be developing a mechanism for ongoing
renewal, growth and adaptation.

**Section 3.4: Analysis of IIS Program Director Workload**

Inasmuch as we heard PDs comment on their increased workload, we felt it
appropriate to explore this issue in more detail. Herewith, then, is our
analysis.

Program proposals were steady over the review period. (One program
director reported a significant increase in 2003.) Nevertheless, the
concentration of ITR grants in IIS led to a problematic increase in
workload. IIS workloads were much heavier than some other NSF program
officers report.

Recent rapid expansion of departments and schools of computer science,
information, and related disciplines brings more researchers into IIS
areas. Changes in grant management in other agencies could make NSF
increasingly attractive. Accordingly, IIS workload should continue to
rise.

Unknowns include the follow-up to ITR, governmental and foundation
response to the cyberinfrastructure report, and imminent reorganization
involving CISE. If the funding that went into ITR is maintained in some
form with the IIS budget explicitly increased, a case for increased
staff and support is easily made. If funds continue to be formally
allocated at higher levels and awarding them continues to be an IIS
responsibility, the need to address workload remains. (In contrast, if
the ITR funding disappeared without being replaced, workload might
return to reasonable levels without intervention, although proposal
levels would have to be monitored.)

Increased staff, technology support, and other resources is warranted
and should help. However, conflicting pressures and incentives must be
managed. Otherwise it may be like adding highway lanes that draw more
traffic and do not reduce congestion.

Program officers would like to be able to give larger awards, but more
than one reported decisions to favor more, smaller awards over fewer,
larger awards \-- to build a research community or to avoid
disappointing too high a proportion of researchers. It is worth noting
the effect on workload: Success in either or both of these legitimate
goals will lead to yet more proposals. Proposals for less money still
must be reviewed, and if less care goes into preparing them they may be
even more difficult to prioritize.

Conversely, if the size of awards is increased, either by making fewer
or through increased funding, the number of proposals could go up with
greater effort going into each one. This could lead to higher rejection
rates and more disappointed researchers.

Finally, if proposal and/or award count is used by NSF to measure a
program's health or success, there is a risk to reducing or holding
steady the number of awards, particularly if additional resources have
been directed to the program. It could appear that more money did not
lead to more product. If proposal numbers drop due to disappointment
with very low (e.g. 5%%-10%) award rates, the program may then appear
unhealthy. If over time the award amounts go up and proposal numbers go
down, the causal direction may not be obvious: it might be suspected
that the awards went up because fewer proposals were coming in rather
than vice versa.

In conclusion, any changes undertaken by a program or by IIS in striking
this balance need to be thoroughly understood and accepted at all levels
of NSF administration. This includes different paths taken by programs
within a division -- it may make sense for one program to build
community through smaller grants leading to more proposals, while
another program increases award size leading to fewer awards and over
time fewer proposals. Two such programs could be equally successful
despite contradictory quantitative measures.

Can we get the best of all worlds -- fewer proposals, larger awards,
greater quality? Perhaps. Writing program descriptions more carefully to
focus on key areas could reduce proposals that are less likely to
succeed. This risks conflicting with the goals of openness and broad
coverage of scientific research, and requires assumptions about the
proper direction of a field. Nevertheless, topic pruning deserves
consideration. Some topics were once important but are less likely to
have an impact; pruning these requires initiative because research
programs have inertia. Other research topics may already be handled by
other agencies or by industry -- in some cases corporate research labs
or even advanced development groups have dozens of highly qualified
people working on problems identical to those tackled by a small
university team.

Workshops of program officers and researchers are given the charter to
shape future directions. They must be constructed to not draw too
heavily on those invested in the status quo.

Clearer program definition, perhaps identifying what is not included as
well as what is included, could also help to reduce proposal migration.
There can be benefits to overlap, but some program officers seemed
uncertain about it, and there are clear problems when a program funds
researchers who do not publish in the area being funded. Another benefit
to clear demarcation of NSF programs is to reduce duplication of efforts
in programs of other agencies.

**Section 4: IIS Programs: Issues and Recommendations**

In this section we examine specific IIS programs in more detail with an
eye towards identifying issues impacting the review process and making
recommendations to address those problematic activities.

**IDM**

IDM has had a long and distinguished tradition within IIS, bridging
traditional CS topics (e.g., database theory) to some of the most modern
issues to face the field (e.g., data-mining from WWW resources). Two
changes from the past were observed during the period of COV review.
First, the fact that the IDM Program Director, Maria Zemankova, has had
such a long-term relationship with the community made her aware of needs
that many established investigators had when vagaries in (e.g., ITR)
funding distributions passed them by. Her response was to distribute
available resources across larger numbers of recipients. The consequence
was that proposals were sliced thinner than might be ideal, but some
level of support was provided to many. (From 1999 to 2002, the
acceptance rate has remained near 20%, but only because the average
award went from \$327,000 to \$224,000.) The other noticeable change had
to do with some minor inconsistencies popping up in the review process
during the absence of Dr. Zemankova (while on assignment to National
Library of Medicine) and the transition to Bhavani Thuraisingham as
interim PD.

The program seems well-poised to build on its past success towards the
challenges posed by changing demands such as the CyberInfrastructure
Initiative. IDM has been exemplary in its support for data-mining and
knowledge discovery programs, connecting the inductive methods of
machine learning to the deductive methods used by databases. New awards
are already addressing key technical challenges such as the integration
of (traditionally logical) database techniques with (typically
probabilistic) unstructured text and semi-structured multi-media
methodologies. Significant interactions between IDM and the newly formed
Human Language & Communication (HLC) program can also be anticipated as
the massive linguistic corpora and computational methods at the heart of
HLC overlap with traditional IDM concerns in information retrieval and
ontology development.

**Robotics and Computer Vision Program**

The Committee concurs with NSF that the change of the name from
\"Robotics and Human Augmentation\" to \"Robotics and Computer Vision\"
is an excellent move. This is in fact a de facto change, since for the
past years proposals and grants in this program have clearly been in two
major categories: Robotics, and Computer Vision. Aspects related to
\"Human Augmentation\" can be handled by other IIS programs such as HCI
and UA. However, there is a significant increase in activity in
human-robot interaction (HRI). Rather than transfer this activity to the
HCI program, we suggest that it either be claimed entirely by RCV, or
shared with HCI.

The research topics in Computer Vision supported by RHA have been driven
by the research interests of the CV community. The Program Director is
most laudable in having taken pains in pooling the opinions of many
researchers in the community to come up with the statement for the new
Computer Vision part of the RCV Program.

The COV reviewed the list of grants awarded and declined in 1999-2002
and a number of randomly selected jackets for individual proposals.
Based on this analysis we conclude that:

-   There appears to be a good balance between awards to established
    investigators and new, young researchers. However, it is our
    impression that a majority of awards are made to established, senior
    investigators. This may be due to the fact that they write better
    proposals, rather than any prejudice on the part of the panels, but
    we do not have sufficient data to verify this statement. We also
    note that many proposals from senior investigators have been
    declined.

-   In a number of jackets we noted that the Panel Summary of a proposal
    is so general as to be nearly useless as feedback to the PIs,
    containing phrases like: "This is a strong proposal with few obvious
    weaknesses", or "This proposal comes from a competent investigator
    and is likely to succeed". Such summaries are so generic they could
    apply to any successful proposal. We suggest that PDs discourage
    their panels from submitting such content-free summaries, since they
    are not helpful to the PIs.

Finally, there is a major national increase in research activity in
these fields, which is translating into a major increase in the number
of proposals received by the RCV program (an increase of 50% from FY02
to FY03). The Program Director proposes to handle this increase by
reducing the size of the awards to the \$70K-90K range. We discourage
this solution, since awards in the range are too small: they do not
allow for support of more than one student, they may not allow
sufficient funding for hardware purchases and may force the PI to spend
time and energy searching for additional summer support.

**DST Program**

The DST Program Director has shown flexibility and the ability to adapt
to change as the scope of the DST program has evolved. In testimony to
the respect with which this program is regarded by the scientific
community, the DST portfolio demonstrates an innovative and
forward-looking core of research projects. The program is
interdisciplinary and covers diverse research areas spanning several
communities of researchers. A lack of proposals from new investigators,
thought to result from this interdisciplinary emphasis, was perceived by
the Program Director as a problem that she would like to remedy. Some
specific strong points of the DST program include its relatively high
percentage of SGER funded work, the use of European panelists, the
emphasis on value-sensitive design, and the support for workshops
related to DST topics.

**Digital Libraries Program**

The Special Programs (Digital Libraries) program appears clearly
different in profile from the regular programs in IIS. It has sponsored
a significantly larger proportion of projects over \$100,000 per year
and a larger proportion of projects under \$50,000 per year, but a
smaller proportion of those in between. Its focus has been large
projects in building digital libraries, a significant number of which
have nothing to do with NSF\'s mission of science and engineering. For
instance, it has sponsored work on a philosophy encyclopedia, a music
library, a library of Jewish folk literature, a library of Native
American culture, a library of Chinese documents, and a library of
cuneiform tablets. Since the program had a 13% rate of funding proposals
submitted over the four years 1999-2002, this raises questions as to the
justification of its allocation of funds.

Here are several concerns we identified:

-   Unfortunately, the criteria for deciding awards are often not stated
    clearly in the random sample of 20 folders (\"jackets\") that we
    examined. Two of the 10 declines were the top-rated proposals on
    their panels as per the average of the assigned reviewers, yet were
    not funded; one of the 10 accepts was rated ninth of 20. Reasons for
    these decisions were not stated clearly in the letter from the
    Program Director. Apparently the panel and the Program Director felt
    diversity in awards (balance in the \"portfolio\") was important.

-   Three of the awards were very small (\$5,000, \$8,000, and \$10,000)
    which is poor cost-effectiveness.

-   The SGER proposals did not appear to be especially innovative,
    contrary the justification for the SGER program.

-   No monitoring mechanisms were in place to confirm if, in fact, the
    funded research did provide a useful digital library for some
    community.

-   In a DLI Phase 2 panel review, 20 or so proposals were ranked
    ordered by the panel. The top proposal was not funded. The 2nd and
    3rd were funded. (Only two proposals were funded.) In the note from
    the Program Director to the rejected top ranked PI: Because of the
    funding limitations, your proposal were judged to be not
    sufficiently competitive (not the exact words). PDs can exercise
    authority and discretion; however, their feedback to the PIs needs
    to be more informative.

We suggest that the Digital Libraries program needs refocusing on the
underlying science and engineering issues to be more consistent with the
other programs is IIS. The technology of digital libraries is by no
means mature, as has become apparent in several of the specific
libraries funded. Refocus is especially important since the Digital
Libraries program is no longer being co-funded by other agencies;
co-funding from them and other parts of NSF for useful digital libraries
should continue to be sought.

**Universal Access Program**

The splitting off of UA from the HCI program a few years ago reflects an
important new emphasis on the need of persons with disabilities and
other traditionally underrepresented groups to participate in the new
information society. Statistics show that as many as 20% of the people
in the United States have some disability. These numbers are smaller for
younger members of the population and increase dramatically with age. It
is clear that the needs of these users should not be ignored if there is
to be full participation in our society. By it very nature, there are
cases in which relevant research will overlap with the missions of other
IIS programs. The Program Directors appear to work well together as a
team to eliminate duplication of effort and jointly fund research when
appropriate. It is recommended that this continues.

The budget for UA is the smallest of any of programs. While this limited
budget, of necessity, results in fewer awards than other IIS programs,
it is encouraging to see that UA has a greater distribution of award
sizes than most of the other programs. This suggests that more so than
the other programs the awards in UA are more likely to adequately cover
the costs needed to conduct the funded research. It is recommended that
this policy continues. Although the funding has grown since the
inception of the program, this budget remains significantly smaller than
that of other programs. The number of submission to UA doubled in 2003.
If this trend continues, the budget will not meet the program needs.

Consistent with the general recommendations of the COV related to
research dissemination, it would be desirable to see the results of
UA-funded research more widely disseminated. Specifically, it was noted
in the COV review of UA awards that many of the PIs do not come from the
traditional UA or HCI communities. This is not necessarily negative and,
indeed, suggests that the UA program is being infused with technology
from a number of disciplines that could significantly impact UA. Given
that the awarding of UA funds addresses specifically issues related to
accessibility, it is recommended that there be an attempt to disseminate
results of these awards widely within the UA community so that results
can readily be applied to address the needs of the intended users. It is
suggested that the UA program holds PI workshops in conjunction with one
or more of the UA community conferences mentioned by the Program
Director (i.e., ACM ASSETS, Closing the Gap, CSUN, and RESNA) with and
eye towards the PIs participating in these conferences.

**HCI Program**

In reviewing HCI Program coverage we consulted the on-line HCI
Bibliography ([[www.hcibib.org]{.underline}](../www.hcibib.org)). This
heavily-used (over one million searches) public index of over 24,000
records covers the full contents of 14 journals, 20 conferences, books
and other materials. It lists 506 authors with ten or more publications.
No PI for the 10 randomly selected FY1999-FY2002 HCI Program awards is
on this list. In contrast, 3 of 10 DST PIs are on the list. Scanning the
full set of awards further indicated that the HCI program grants are not
fully reflective of the HCI literature.

We also reviewed panel members for FY 2001-FY2002. DST panels included
two to three times as many established HCI researchers (measured by HCI
publication) as the HCI panels.

The HCI field emerged from two traditions. Post-WWII human factors or
\'man-machine interaction\' research emphasized tool use by operators
who had little discretion. Personal computers and workstations led to
more discretionary use of software. The HCI program has emphasized the
former: technologies such as speech or brain wave control of displays
whose users to date have no discretion, due to disabilities, hands being
engaged, and so forth. The new program description does a good job of
identifying emerging technologies, but under-represents basic research
in design and interface assessment methods that are central to
discretionary software and have occupied much of the HCI literature
since the early 1980s.

The Human Languages and Communications Program focuses on situations
involving less discretionary use. This creates an opportunity for an HCI
program that covers more of the topics addressed by HCI groups in
Computer Science and Information departments and schools, as reflected
in major venues such as the ACM CHI conference.

Below is a draft of the HCI Program mission as it might be revised to
include additional topics.

=====

**HCI Program Synopsis**

The HCI program supports research fundamental to the design and
evaluation of systems that mediate between computers and humans, and
which will lead to the creation of tomorrow's exciting new user
interface software and technology.

The program's ultimate objective is to transform the human-computer
interaction experience, so the computer is no longer a distracting focus
of attention but rather an invisible tool that empower the individual
user and facilitates natural and productive human-human collaboration.

HCI research topics include[^2], but are not limited to:

-   ***Development and evaluation of new design methods***

-   Development and formal experimental evaluation of foundational
    models and theories

-   ***Information architecture***

-   ***Augmented cognition (aids for various cognitive tasks)***

-   ***Information visualization***

-   Multi-media and multi-modal interfaces

-   Intelligent interfaces

-   Virtual and augmented reality

-   Immersive environments

-   Wearable, mobile and ubiquitous computing

-   ***Collaboration technologies***

-   New I/O devices

Because such research is highly interdisciplinary in nature, HCI program
interests often overlap in part with those of other programs within IIS
Division and across NSF.

=====

As noted in our review of the Universal Access Program, basic speech and
language research funded there that could contribute to technologies
directed at universal access is sometimes not directed to such
audiences. Language research is also funded in IDM. We recommend that
proposals for language research that have gone to HCI, UA, and IDR
should be directed to the new Human Languages and Communications Program
except where their primary focus is on multimodal interfaces, universal
access, or information retrieval. The perceived great potential and
recognized great difficulty of speech and language research have shown
that resources need to be carefully managed to avoid duplication and
waste, and preserve a balanced research portfolio.

[^1]: Subchapter 300-Committee Of Visitors Reviews, 1/2002.

[^2]: The bold and italicized bullets are proposed additions to the
    program synopsis.