###### Committee of Visitors FY00-02 Report

###### National Science Foundation

###### Division of Advanced Networking Infrastructure and Research (ANIR)

###### May 8 & 9, 2003

###### Executive Summary

A Committee of Visitors (CoV) convened May 8 & 9, 2003 to review the
programs of the Division of Advanced Networking Infrastructure and
Research (ANIR) for FY00-02. The Division groups together two primary
activities: research related to advanced networking technologies
(including the Internet and its successors) and the funding of advancing
networking infrastructure. This latter activity sometimes requires
establishing connections for secondary schools, higher education, and
international projects.

The Division's efforts are under serious stress because of three
developments. The downturn in the US and world economies (the bursting
of the telecom bubble) has had a very negative impact on the role of
industry in telecommunications research. The demise of many of the
startup companies in this area, as well as severe reductions in the size
of industrial research labs, has put a number of talented researchers in
search of funding, causing in part a sharp increase in the number of
high-quality, fundable proposals to NSF in the networking and
telecommunications areas. To compound matters, other government agencies
have seriously reduced their funding commitment to computer networking
research, leaving NSF as one of the few remaining funding sources in
this area. This increase, in combination with the announcement of new
NSF programs -- such as Information Technology Research (ITR), is
reflected in a more than doubling of the workload for program managers
and administrative staff in ANIR. This exacerbates the concern expressed
in the last CoV report regarding attracting and retaining program
managers, the proposal review process, and the follow up post-award
process. The ANIR staff has risen admirably to the challenges presented
by the rapidly growing workload, but such heroic efforts are already
taking a toll and cannot continue indefinitely without seriously
compromising the programs of the Division.

As part of the CoV process, the Committee reviewed ANIR proposal jackets
(both awarded and declined), overall summaries of the programs,
aggregate data provided by the Division, and held frank discussions with
the Division staff. The Committee feels that, overall, the programs are
well run and are producing very good value for the dollars invested. The
research outcomes are valuable to the scientific community and society
as a whole and deserve praise. The infrastructure deployed is bringing
value to both the education and research communities, and transfer of
technology appears to be successful.

The CoV has the following primary findings and recommendations:

-   **Information Technology Research**. The ITR initiative has had a
    very positive effect on the Division's research program, and it is
    yielding admirable results. Medium-size ITR projects focus on
    important systems-oriented research problems, and the small-size ITR
    projects pursue timely research that embraces higher-than-normal
    risk. The Committee feels that this freedom to explore higher-risk
    problems is not only very important for leading advancements in the
    ANIR areas of interest, but it is also essential for our nation. The
    degree of collaboration fostered by the program is also noteworthy.
    In particular, Committee members felt that the medium-size ITR
    projects achieved much needed interdisciplinary collaboration
    allowing substantial effort to be applied to important problems that
    would otherwise go unexamined. The Committee assessed only two
    relatively new large size ITR projects. The Committee finds it too
    early to draw firm conclusions based on these two sample points, but
    the topics being considered, the techniques being applied, and the
    teams' composition were appropriate.

*The Committee recommends that the ITR program be evaluated for
direction to keep it in the "high-risk" research track, but continued in
some form, since it provides an essential and unifying framework, as
well as a high return on investment.*

-   **Infrastructure vs. Research**. The Committee finds that the
    percentages of funding allocated towards infrastructure and
    research, as recommended by the previous ANIR CoV report have been
    met or exceeded, and that the trend to reestablish research funding
    on at least an equal footing to infrastructure should continue. The
    Committee further believes that networking research and
    infrastructure projects are complementary, reinforcing each other in
    a cycle of development, testing, and deployment. The Committee
    therefore strongly recommends that any NSF networking activity
    continue the tradition of tight integration of infrastructure with
    the associated research. However, the Committee believes that it is
    critical to bridge the gap between research in experimental networks
    and research in real networks. The Committee further stresses the
    need for NSF to help researchers to obtain actual data and
    measurements from real, often commercial networks, and that this
    goal is critical not only to national security but also to the US
    economy. Serious concern was expressed that a significant amount of
    community outreach falls under the province of ANIR. While these
    endeavors are important to the mission of NSF, they distract from
    the equally important task of funding networking research, as a
    vital and necessary science in and of itself. The need to attend to
    operational details of deployed infrastructure sometimes siphons
    talent, investment, and time away from the research mission.
    Therefore, it is of the utmost importance that the NSF continues to
    identify and nurture work on outstanding network research issues, as
    they have the potential to have an impact on all scientific
    disciplines that rely on networking infrastructure.

*The Committee therefore strongly recommends that any NSF networking
activity continue the tradition of tight integration of* *programs that
advance network infrastructure with the associated research* *in
networking.* *Networking research demands some of its own
infrastructure, which may be shared with deployed infrastructure, and
the control of this infrastructure is best vested with the networking
research community.* *Furthermore, projects in* *instrumentation, data
collection for research purposes, and measurements in real-life networks
(where possible)* *must* *be given high priority.*

Research and infrastructure activities mutually interpenetrate each
other, forming a continuum between pure research and deployment but
often combining advanced ideas with experimental prototyping. The
interplay between research and infrastructure is reflected in the
oft-stated observation that "the next generation Internet will be built
as an overlay on the current generation of the Internet".

-   **Middleware and Grid**. The NSF Network Middleware Initiative (NMI)
    program has been very successful in highlighting a set of important
    issues. On the one hand, it has drawn considerable attention to the
    lack of standard middleware for distributed computing and for
    end-to-end communication environments. On the other hand, it clearly
    illustrates how difficult it is to define a universal middleware
    software stack. The Committee noted a certain level of mixing of
    goals between Grid computing initiatives, middleware initiative, and
    possibly the forthcoming Cyberinfrastructure thrust. It also has
    noted the NMI actually has been quite successful in stirring
    interest from industry, which appears to have taken the lead in
    development of the middleware stack. As such it has been successful
    in effecting transfer of technology to the industry. However, it was
    also noted that there appears to be a lack of institutional
    diversity within NMI awardees, and that the actual software product
    appears to be overly complex and unstable, resulting in an apparent
    lack of successful adoption of the product by a broader customer
    base. While research projects can provide excellent proof-of-concept
    and prototype implementations., this raises the question as to
    whether NSF should be in the business of funding robust production
    software − something that full-time open source developers or
    commercial developers may be much better at, and that in this case
    NMI funding may now overlap with investments made by industry, other
    government agencies, and other areas of NSF. In light of large
    investments being made by computer vendors, the impact of NMI
    related NSF funding is sure to be diminishing, except in perhaps a
    few well-chosen areas. A clarification and unification of the
    directions is desirable. As a result, the Committee expects
    Cyberinfrastructure-related initiatives to unify networking, Grid,
    NMI, and similar efforts.

> *The* *CoV recommends that the NMI program be reevaluated for
> direction and scope in the light of ongoing commercial efforts and
> related initiatives in NSF and other agencies, especially in the
> context of the emerging Cyberinfrastructure initiative.*

-   **Staff Workload**. With the increase in the staff workload -- as
    evidenced by the more than doubling of the incoming proposals over
    the period of FY00-02, and with the expectation of another doubling
    in the FY02-03 period (and even more as the Cyberinfrastructure
    program ramps up), the Committee feels strongly that, unless
    corrective measures are taken as soon as possible, the quality of
    the research review and award selection process, and by implication,
    the return on investment of the NSF funds, are at risk. Given the
    current numbers of program officers and administrative staff, the
    workload of developing new programs, justifying existing programs,
    organizing and supervising panels, and selecting and administering
    awards has risen to an unacceptable and clearly unsustainable level.
    The heavy workload places such a taxing burden on the program
    officers and the administrative staff that there is a real risk that
    serious, damaging, and embarrassing errors might be made in the
    delicate process of merit-based peer review.

The CoV recommends that the staff overload situation be addressed
immediately. An increase in resources or a change in processing steps to
decrease workload is immediately necessary.

Other areas of concern and note include:

-   **Panels**. The current practices of panel selection, preparation
    for panel meetings, and documentation of the process could be
    streamlined without sacrificing the quality of the merit-based
    peer-review process, more efficiently utilizing program officer,
    administrative staff, and panelist time.

> *The* *CoV recommends that the panel selection, preparation, and
> documentation process be re-evaluated and streamlined.* *Changes in
> panel practice to increase risk-taking outcomes should be encouraged.*

-   **International Network Connections**. The CoV acknowledges the
    importance of international connectivity to scientific
    investigation. We acknowledge that entities like NSF and their
    overseas counterparts need to take the initiative in deploying and
    managing these facilities until the beneficiaries can incorporate
    the services into their operating budgets. The program has made
    great strides in connecting North America, Europe, and Asia. Yet
    there are still elements of the global scientific community that
    need to join their infrastructure with ours (.e.g., South America).
    The CoV firmly believes that this will result in upgrades to the
    local infrastructures, further leveraging the funding.

> *The* *CoV recommends that NSF consider partnership* *support for
> high-performance connectivity for scientific collaboration with
> communities that may not* *yet* *be in the "loop", such as Central and
> South America. Note that* *Trans-American high-performance Information
> Highway (or Cyberinfrastructure) may have not only extensive positive
> scientific implications, but also highly desirable economic ones.*

-   **Integration**. The Committee saw evidence of good collaboration in
    many of the programs, but especially in ITR. Nevertheless, there was
    occasional skepticism about the value of some collaborations,
    especially extremely large ones, since it was not always clear that
    collaborative integration has taken place, and that team members
    were adequately motivated to work together.

*The* *ANIR* *CoV* *recommends that the ITR CoV take this matter under
consideration* *and define new means and measures for collaboration.*

-   **Cooperative Agreements**. The post-award follow-up on
    cooperative-agreement awards raised questions for the Committee in
    terms of how effectively some projects were being tracked and
    administered. Given the requirements for a more proactive style of
    management imposed by cooperative agreements, it was unclear from
    the provided documentation how well some programs were being
    administered to achieve their stated goals.

The Committee recommends that program officers employ performance
metrics, better reporting, and clear documentation to track the
achievement of the goals of the programs in which cooperative agreements
are the dominant award vehicle. To encourage management improvements, a
Division meeting where a minimum one-page summary four times per year
for each Cooperative Agreement might be considered.

-   **Committee of Visitors***.* *Finally, the Committee recommends
    several improvements that could be made to the CoV process itself.*

Members found it difficult to complete the CoV process as it
traditionally has been conducted. The process historically has centered
around the review of a sample of the Division\'s proposal jackets, which
is a painstaking effort that members felt gave only a partial picture of
the overall state of the Division\'s projects. Although this process
allows members to evaluate individual proposals, this makes it difficult
to answer questions about a specific panel or an entire program, since
this requires CoV members to wade through a large collection of related
jackets in the short amount of time allotted to the review.
Specifically, better panel-wide and program-wide information must be
made available to the CoV, and the Committee should have the ability to
ask for this data (e.g., summary reports) in near real time. Although
the Division provided a large amount of data and reports to the CoV and
made valiant efforts to compile specifically requested reports in a
timely fashion, this process still was viewed as less than optimal; a
data warehouse that contains jacket information and that may be queried
by Committee members or NSF staff would simplify greatly the CoV
process. Such a setup would considerably improve not only the quality of
information available to the CoV, but also the efficiency of the time
spent both by CoV members and NSF staff. It was also noted that the
Committee might be better utilized if a small subCommittee were to have
a pre-meeting a few weeks in advance of the actual CoV meeting to
identify the specific reports needed and to make this information
available to members in advance of their assembly at the final meeting.
Detailed suggestions to improve the CoV process are made in Section 3
and in Appendix C of the CoV report.

###### 1. Introduction

The members of the CoV were:

1.  CoV Chair, Dr. Joseph A. Bannister, Information Sciences Institute,
    University of Southern California

2.  CoV Cochair, Dr. Ron Hutchins, Georgia Institute of Technology

3.  Mr. James R. Bottum, Purdue University

4.  Prof. Anthony Ephremides, University of Maryland, College Park

5.  Dr. Alan Huang, Terabit Corporation

6.  Dr. Sri Kumar, Defense Advanced Research Projects Agency

7.  Dr. Bob Laddaga, Massachusetts Institute of Technology

8.  Mr. David M. Meyer, Sprint

9.  Prof. J Strother Moore, University of Texas, Austin

10. Prof. Richard R. Muntz, University of California, Los Angeles

11. Dr. Craig Partridge, BBN Technologies

12. Dr. Eve M. Schooler, AT&T Labs - Research

13. Prof. Mladen A. Vouk, North Carolina State University

Dr. Bannister is the representative of the CISE Advisory Committee, of
which the CoV is a subcommittee. The Division was last reviewed in June
2000. **Specific actionable recommendations are scattered at the
appropriate places in this report, and are highlighted by
[underlining]{.underline}.**

###### 2. Division-Specific Issues 

The CoV met initially as a group and then split into three groups to
review ANIR's programs. The groups were tasked with analyzing the
Network Research, Information Technology Research, and Network
Infrastructure programs. Once underway, however, Committee members noted
that it was sometimes difficult to categorize neatly the different
program elements. Over the years there has been a mixing of purposes
that sometimes obscured the original intent of the different programs.
Thus, it was observed that Network Infrastructure embraced some projects
that might easily have been classified as research, or that Information
Technology Research projects sometimes appeared to fit more readily into
the Network Infrastructure program. It was observed that some of the
boundaries between programs have blurred, and that this situation has
made it less obvious to the community what programs were topically
appropriate. While recognizing the legislative and administrative causes
of ANIR's current programmatic structure, the Committee thought that it
would be simpler to unify the programs or reconstitute them along
different lines. Indeed, some CoV members felt that a broad program
should be established that seeks to invite a diverse collection of
proposals with minimal guidance as to which topics or research
modalities are being encouraged during a solicitation.

###### 2.1 Future of the Information Technology Research Program 

There is no doubt that the Information Technology Research (ITR) program
has enhanced the research sponsored by ANIR. All three award levels
(small, medium, and large) of ITR projects are found within ITR, and
their funding represents a significant portion of the Division's budget.
This program has increased notably the amount of inter- and
cross-disciplinary collaboration supported by the Division, opening new
avenues of investigation for many in the network-research community. The
element of collaboration, coupled with support for larger projects, has
also promoted the expansion of systems-oriented projects with richer
experimental components.

Building on its long tradition of intradisciplinary collaboration, the
network research community took readily to the opportunities offered by
ITR. The Committee saw ample evidence of collaboration not only within
the networking discipline but also between networking and other
disciplines. Such collaborations were seen to strengthen the field by
promoting the synthesis of new ideas, integrating disparate scientific
approaches, and extending networking to new application domains. The
benefits of this aspect of the ITR program are compelling and support
the argument for continuation of the program or the inauguration of a
new program with like objectives for collaborative research.

The benefits of collaboration should be weighed carefully against the
tendency to

over-emphasize the application of networking at the expense of the
fundamental science of networking. The CoV questioned whether projects
were being funded largely on the basis of the attractiveness of
applications rather than on the intellectual merit of the underlying
ideas. While it is tempting -- even for expert panels -- to recommend
awards to projects that have articulated exciting new applications, the
Division must maintain the discipline of promoting work that advances
networking science.

Given NSF's goal of supporting science and engineering, it is not
surprising that ITR awards are concentrated in the area of e-science and
support for distributed scientific collaboration. Moreover, the fields
of science and engineering that are most prominently represented are
usually "traditional" sciences, such as physics, geology, and biology.
Outreach to other areas was not as evident to the Committee. Even
outreach of networking to CISE's computer science constituency was
problematic. The Committee naturally worried about the notion that the
intrinsic value of computer science comes exclusively from its role as a
tool for "real" sciences. It is, therefore, imperative that networking
be viewed as an activity that can be harnessed to serve not only a
narrow spectrum of science but also other branches of science, such as
engineering, social sciences, medicine, and even computer science
itself. Although strong affinities exist between network researchers and
scientists from other branches, it should not be the case that all
collaboration should be restricted to embrace only those in the
scientific domain -- the Division should encourage all networking
research that serves society as a whole, not closing off pathways that
might benefit the broader spectrum. As our collective experience has
shown, the Internet's focus on general-purpose modes of communication
ultimately benefited the community of scientists more than had early
networking pursued a shorter-term, single-minded goal of making linking
together only highly capable supercomputers.

Over the years ANIR has made great strides in advancing the stature of
networking research. A balanced approach that combines theory, systems
research, and infrastructure deployment and testing has helped
networking to progress to its current state of development. Yet, there
is the sense that networking researchers are so consumed by the
relentless drive to design, develop, and deploy new technologies and
protocols that the community has not had the opportunity to characterize
their creation adequately, either in terms of understanding the
underlying phenomena of network traffic, the structure of the Internet
topology, or the significant patterns to be found in network
measurements. Indeed, one may find examples of work that succeeds in
bettering our fundamental understanding of network processes and
structure, but these instances are relatively rare. The Division is
attempting to promote what has been called the science of networking,
organizing workshops to delineate strategies for establishing a research
agenda and recommendations for bringing this agenda to life. It is clear
that a science of networking must rely on data and observation. It is
thus critical to its success that a precondition for the new agenda
include a thoughtful effort to collect data about today's Internet. This
plan has -- to a limited extent -- already been set in motion by ANIR,
but a more comprehensive activity is required. This sentiment is echoed
in a report[^1] of the Computer Science and Telecommunications Board of
the National Research Council that urges scientists to document "a day
in the life" of the Internet, in order to provide an empirical
foundation for the development of deeper models of networks and their
processes. Such a new science of networking draws from the traditions of
both theoretical and systems research and must involve collaboration
with infrastructure planners and operators. ANIR should prepare itself
to deal with the technical and nontechnical challenges that accompany
the implementation of this program.

###### 2.2 Networking Research and Infrastructure

Because the NSF has limited funding dollars, a perennial debate exists
about the proper balance between funding infrastructure versus funding
research. On the recommendation of the previous ANIR CoV, the percentage
of funding for infrastructure was adjusted to about 50% of the budget in
the years reviewed and about 30% in 2003. The Committee agreed that this
is an important trend that should continue, or at least should not be
reversed, as the renormalized ratio is seen to provide better interplay
between research and infrastructure and is closer to the Foundation's
overall target for the ratio (NSF currently maintains a ratio of about
22%, but higher ratios are under consideration[^2]). Since
infrastructure is becoming more tightly coupled with actual research --
which is an excellent trend -- it may be difficult to classify properly
the different programs into strictly one or the other category. For
example, cases in point are the new EIN and NRT programs, which
reinforce both the research and infrastructure components of ANIR.

The committee acknowledges that there is a delicate interplay between
Infrastructure and Research. Everyone recognizes the need for the
networking research community to be involved with real networks, meaning
infrastructure on which to deploy experiments and test research
theories. However, the ongoing debate centers around where to draw the
line between spending funds on experimental networks versus on
production networks. To draw persuasive conclusions, beyond small-scale
prototyping and proof-of-concept state, researchers require access to
production networks (or at least raw production traffic and performance
data), i.e., networks with real Internet topologies, with real routing
policies, with real economic pressures, with real users, and exhibiting
real every day challenges faced by ISPs, end-users and so on. Thus, the
CoV noted that it is questionable how useful strictly experimental
networks can be for some research when it comes to assessment of
scalability issues, or large-scale effects that appear in complex
real-life systems. It is therefore essential that network research be
paired with adequate infrastructure and tools to support experimentation
and demonstration.

The CoV strongly recommends that the NSF promote research efforts that
help bridge the gap between research and reality. One reality is that
much of the Internet infrastructure is owned by commercial entities,
thus the CoV also urges the NSF to work to foster and to jointly fund
cooperative research endeavors between government and industry. The
danger of ignoring network realities is that the research conducted
strictly in experimental testbed environments risks being deemed
irrelevant.

Along the same lines, the CoV recommends that the NSF push for access to
real traffic data from deployed commercial networks and further
recommends that network measurement be a strong component of all network
research. Actual data and measurements are vital to national security
and the nation's future economic health. They have the potential to
expand our understanding of the real Internet infrastructure, which in
turn would allow us to defend against and to respond to various natural
disasters and malicious attacks, to improve our ability to scale the
current network in terms of size and traffic, and to gain insights into
how best to accommodate new high bandwidth applications and how to merge
the telephone, cable, and data networks.

Another serious concern regarding funding dollars for Infrastructure vs.
Research is that a significant amount of community outreach falls under
the province of ANIR. For instance, the ITR can be viewed as an outreach
to other scientific disciplines such as Astronomy, Physics, Biology,
Geology, etc., as well as an outreach to new applications. The Grid and
Middleware can be viewed as an outreach to the computer hardware
community as well as an outreach to the application level software,
real-life IT based workflows of critical nature, and so on. While all of
these are important endeavors that fall under the mission of the NSF, in
general outreach drains the talent and resources of the **computer
networking community.** As there is no natural end to this outreach, the
CoV recommends that the NSF establish criteria to determine where more
precise lines should be drawn, both in the funding and the management of
other communities\' network-related projects. One solution would be for
NSF to provide supplemental funds to ANIR to assist in network-related
outreach programs. Another solution would be to make outreach a
significantly smaller component of ANIR.

Most importantly, the CoV recommends that ANIR more fully recognize that
there is real science in Internet architecture and networking research.
Therefore, it is of the utmost importance that the NSF identify and
nurture work on outstanding network research issues. Until these issues
are addressed, network infrastructure will continue to require large
operational costs and will continue to be marginalized, in terms of the
utility it provides to the more traditional sciences.

The most recent workshop on Outstanding Issues in Network Research
(http://www.cs.virginia.edu/\~jorg/workshop1/infotoparticipants.html) is
one example of a successful technique to bring expertise to NSF on what
the larger community perceives are the open networking research
challenges. A series of similar workshops on open research issues might
be warranted, to gather ideas from a wider range of experts.

The strong connection between networking research and the deployment and
operation of real infrastructure is seen by the Committee as sufficient
justification for retaining these programs under the aegis of a single
division or at least to coordinate the activities closely. The CoV
certainly recognizes the challenge of comanaging of these different
activities. Moreover, the boundary between the two activities is in
constant flux, and separating infrastructure from research could produce
inefficiencies by creating both overlaps and gaps. It was seen as
imperative that research infrastructure, such as testbeds and
experimental platforms, be tightly integrated with research activities,
so that research considerations are given full weight in acquiring,
developing, and managing these resources.

While infrastructure is in large measure an enabler for the scientific
disciplines represented in NSF, it is important to note that *all*
disciplines should derive benefit from deployed and developing network
infrastructure. The myth that certain application domains require
network infrastructure more than others is sometimes promulgated by
overemphasis on high-bandwidth connections. The key differentiator is
services and service rates rather than data rates, and infrastructure
efforts occasionally lose sight of this fact. The program should strive
to identify and implement services that can be used by all members of
the scientific community. The Division should therefore identify
specific needs of research communities and engage these communities in a
mode that shares goals, costs, development effort, and successful
outcomes. Too often, networking professionals are viewed by their
collaborators as "plumbers" or "code slaves" required only to erect a
vehicle for a scientific community to perform a fixed number of
experiments. The CoV noted that the field of computer science may itself
not have effectively used networking infrastructure to make advances in
the field. A better understanding of how to make networking more
effectively meet the needs of practicing computer science researchers
has a good chance of revitalizing and expanding the field of networking.

###### 2.3 The NSF Middleware Initiative

###### 

The NSF Middleware Initiative (NMI) program was born right at the time
when scientific and other users of sophisticated networked computational
resources were clamoring for advanced services to be layered atop a
well-developed suite of basic Internet services. Spanning Grid
technology, authentication and authorization services, Internet
conferencing tools, resource discovery, content caching, and remote
system management, the program has contributed substantial software to
the middleware code base. Recent announcements by major computer vendors
to build upon this base validate the foresight and utility of the NMI
program.

The CoV examined a number of issues in the NMI program. The first issue
is the utility of the technologies to the NSF science and engineering
community, especially with respect to the question of whether scientists
and engineers have adopted the technologies under development. The
second issue is how effectively the program is being managed to produce
well-coordinated software releases and well-designed integrated code
with sufficient documentation and training to promote rapid adoption by
the NSF community. Fro its third issue the CoV sought to understand the
Division's strategy for handing over to commercial companies the
responsibility for areas of middleware (e.g. Grid services, which are
touted as a cornerstone of so many companies' enterprise solutions).
Finally, the Committee explored programmatic issues, such as
institutional diversity in NMI, strategic planning, redundant funding of
technologies, outreach to science and engineering communities, and
training of students.

###### 2.3.1 Evolving Environment

Continued global competitiveness is widely believed to depend upon
maintaining leadership in the development and management of new
information technologies[^3]. For example, rapidly changing technologies
have pervaded every sector of American society, infusing nearly
everyone's work and personal lives, that of scientists included. The
efficiency of those already dependent on IT to conduct their daily
activities needs to be increased by making IT more appliance-like. One
component of that is what can be termed distributed computing.
Distributed computing, and ways of improving, facilitating and making it
more efficient and robust, functionally capable, and end-user friendly,
cost-efficient and conforming has been the object of research and
development since the day the networks were invented.

The most recent integrating view of this space is provided in the NSF
blue ribbon report on Cyberinfrastructure[^4] on the future of the U.S.
IT infrastructure -- the so-called cyberinfrastructure, and the
associated computing and networking research directions. To quote (from
appendix): *Cyberinfrastructure makes **[applications dramatically
easier to develop and deploy]{.underline}**, thus expanding the feasible
scope of applications possible within budget and organizational
constraints, and **[shifting the scientist's and engineer's effort away
from information technology development and concentrating it on
scientific and engineering research]{.underline}**. Cyberinfrastructure
also increases efficiency, quality, and reliability by capturing
commonalities among application needs, and facilitates the efficient
sharing of equipment and services".*

What this means, among other things, is that modern scientific studies
and experiments, and other areas where IT is used, can be quite complex
and often produce large quantities of data that require effective and
efficient management. They also require complex and intensive
computational support and analyses. Yet, the task of managing access to
and computational treatment of this information instead of becoming
easier, is becoming so overwhelming that end-users are spending more and
more of their time tending IT itself and developing special purpose
solutions, rather than using their time effectively for scientific
investigation and discovery, or other domain specific activities.

###### 2.3.2 "On-Demand" Services and Resources

While some of the lower level services provided by the networking,
computing, and application IT infrastructure have been standardized, and
thus provide a relatively stable base upon which one can build, this is
not necessarily true with higher level layers, such as services related
to ready provisioning of more complex resources to an end-user, or a
person constructing higher-level end-user services. The vision is that
of "on-demand" application or domain tailored services and resources,
and an appropriately supportive networking infrastructure and
middleware.

![](media/image1.wmf){width="2.875in" height="1.7458333333333333in"}

![](media/image1.wmf){width="2.875in" height="1.7458333333333333in"}

**Figure 1**. Inefficiency of an end-user increases (red cone) the
further the user is from the domain of expertise.

For example, a scientist is using a combination of experimental,
theoretical, simulation-based and computer-based analytical and
decision-making methods to work on a complex problem. The scientist's
workstation is connected to the "Grid". The "Grid" (perhaps incarnated
as an advanced implementation of cyberinfrastructure and enabled through
appropriate networking and other middleware layers) guarantees services
and resources authorized for the project regardless of geographical
location of the project entities and elements, and provides them
on-demand and within the quality of service and information expectations
of the end-user. It is akin to having a "wall socket" that provides, for
example "scientific analysis support". The scientist receives
experimental data from local and remote sources (e.g., from a
micro-array reader), and accesses a number of external storage and data
services and analysis sites in a transparent fashion, amounts, and speed
needed for his or her work. The sole bottleneck is the scientist's
ability, as a human being, to analyze the problem, nothing else. Even
that activity may be enhanced through intelligent agents and
cyber-assistants. To do that, we need a suite of principles, tools and
distributed services that reliably improve productivity of end-users by
providing pro-active on-demand information technology based support for
their specific problem solving workflows.

For example, consider the IT "stack" within which an end-user operates
(Figure 1). It consists of an End-User layer (end-user semantics and
workflows), supporting applications, operating system, communication
elements and finally data storage and computational resources as well as
physical communications. When end-users (such as scientists) communicate
with each other, or indeed just manipulate and analyze data, the most
inefficient IT support currently occurs at this potentially most
rewarding problem solving layer -- the context or domain layer of the
interaction. In fact, most of IT middleware and tools available today
focus on the syntactic layer and below. As one progresses down the
"stack", the information technology tools are more plentiful, but also
more difficult to use by someone whose domain expertise is not
information technology. This needs to be fixed. One of the principal
purposes of various middleware initiatives is to provide and validate
tools that adequately fill inefficiency gaps and minimize the red area
(from the perspective of a higher-level user.).

The "grid" efforts of today, the web services efforts, the terascale
computing initiative, and the ANIR NMI program are part of a suite of
research and development activities that are attempting to explore and
build support elements for such a proactive "on-demand" resource world.

The NSF NMI is a program that emerged with the hearty endorsement of the
previous ANIR CoV. Recognizing the importance of middleware in the
development and deployment of new services, ANIR took on the task of
accelerating research, development, and dissemination of software
tailored to advance scientific computing. Coming at the auspicious
moment when Grid computing was just taking off, NMI has been able to
leverage key pieces of Grid computing and other efforts to produce a
body of code, a "software stack" that implements a number of important
functions.

On one hand, the program has been very successful in highlighting a set
of important issues, and in stirring interest from industry that has now
taken over the development of a certain part of the middleware stack.
NMI has been successful in effecting this transfer of technology.
Successes stemming from this effort include (1) overwhelming industry
commitments towards "grid" middleware, e.g. IBM, HP, Sun, (2) GGF based
on the IETF model, and (3) strong interest by the scientific computing
community.

On the other hand, the NMI effort has also drawn considerable attention
to the lacks in "standard" integrating middleware for distributed
computing and end-to-end communication environments. It is clearly very
difficult to actually define and unify this developing "software stack"
to everyone's satisfaction. For example, it is sometimes difficult to
distinguish between Grid computing initiatives, the middleware
initiative, and possibly the forthcoming Cyberinfrastructure thrust.
There also appears to be a lack of institutional diversity among NMI
funded projects -- that may or may not be a practical problem, but it
may be related to the issue of prototyping versus development of
hardened production software. Finally; the actual software product
appears to be overly complex and unstable, and this may be responsible
for an apparent lack of successful adoption of the product by a broader
customer base.

Researchers have always been good at providing theoretical and empirical
output, and in the software arena, prototypes and proof-of-concept
software and packages. However, the NMI experience raises the question
as to whether NSF should be in the business of funding production
software − something that open source developers or professional
commercial developers may be much better at. Hence, an issue is whether
NMI funding overlaps with investments made by industry, other government
agencies, and other areas of NSF. A clarification and unification of the
directions is desirable. Perhaps the new Cyberinfrastructure-related
initiatives can provide that matrix and unify Grid, NMI and similar
efforts. Some specific concerns include (1) funding of specific pieces
should move to industry, (2) coordination of efforts is too loose, (3)
code is not being used in preproduction or production systems, (4)
funding of some pieces (e.g. Globus/Grid) comes from multiple programs
and it is unclear which program is funding which piece, (5)
acknowledgment of NSF support appears to be poor -- NSF may lose the
opportunity to claim major credit for middleware accomplishments

However, we would like to close this section by reiterating that, in the
opinion of the CoV, the NMI program has already made an extremely
valuable contribution to the advancement of Cyberinfrastructure. It may
be appropriate to re-examine the middleware component of the NSF
interest area and merge it with that of the on-demand goals of the
emerging Cyberinfrastructre initiatives.

###### 2.4 Career and Quality-of-Work Issues

**2.4.1 Developing the Careers of Researchers\
**

The success of the NSF CAREER awards to young researchers has prompted
the CoV to consider useful expansions of the program. The Committee
believes that there would be great merit for ANIR and for the country to
provide **career transition awards** to established researchers who wish
to switch from their current field of study to networking or
network-related activities (OS and security researchers come to mind as
interesting categories).

One caveat with the success of the CAREER program is that the receipt of
a CAREER grant is now perceived as an indicator of tenure worthiness.
Thus, the Committee recommends that the NSF take this point under
consideration, to evaluate if the program can or should be modified to
eliminate this perception. It was noted that CAREER grants provide only
a modest amount of funding dollars, so award amounts should not be
reduced to accommodate additional numbers of awards.

**2.4.2 Developing the Careers of NSF Program Officers\
**

Another very important category of transition award would be one to help
NSF program directors reestablish their research programs at the end of
their NSF tenure. The spirit of community service is nowhere as evident
as it is in NSF Program Officers. Whether IPAs or civil servants,
Program Officers are working scientists who have chosen to make many
career sacrifices in service to their communities. The committee
suggested that a tangible reward structure is appropriate as partial
compensation for their sacrifices. Equally important, incentivizing
Program Officers is a key to recruiting high-quality personnel and vital
to the success of the Division. Perhaps post-service awards could be
made before Program Officers begin government service, but deferred
until they return to their home institution to promote reinsertion of
Program Officers into the research community.

**2.4.3 Support and Staffing\
**

The staffing for technical and business support for ANIR Program
Directors is inadequate. It appears that there is one support person per
program at ANIR, which is far less than what Program Managers and
Directors are provided with in sister agencies such as DARPA. Typically,
DARPA program managers are supported with at least one full-time
technical-support staff (who are often PhDs) to assist the Program
Manager in keeping track of latest scientific and technical developments
and assessing their potential for creating new programs, and at least
one full-time business staff person who takes care of grant and
contractual matters including post-award monitoring. In addition, DARPA
Program Managers have several additional staff members who help in the
proposal-review process. The number of proposals submitted to an ANIR
program is often two to three times more than what is submitted to a
typical DARPA program. This appeared to the CoV to be quite a burden for
both the ANIR support staff and the Program Directors, as was confirmed
in our interviews with the ANIR staff. With the rapid increase in ITR,
and ANIR's prominent role therein, the load on the staff has become
worse. Given that this COV report is also recommending that ANIR Program
Directors do increased post-award work on follow-up and monitoring, the
situation becomes even worse. To relieve the increasing burden on
Program Directors and support staff, the Committee recommend that
support staff be increased, with at least two support staffers for each
Program Director.

###### 2.5 Improving the Review Process

It was noticed that the quality of the proposal reviews varied widely.
The CoV has suggestions on how to improve this situation. The suggested
themes of "intellectual merit" and "broader impacts" have helped focus
the review process. Adding some additional theme such as creativity,
risk, time scale of applicability, benefits, breadth of applicability,
cost, and the examiner\'s familiarity with the subject would help add
more structure to the reviews.

It was noticed that there was no easy way to compare reviews. How does
an "exciting" idea compare to a "truly significant" idea? It was also
noticed that there was no easy way to compare the various proposals. How
does a "very good", "fair", and a "poor" for one proposal compare
against a "very good", "good", and a "fair" for another proposal? It is
also noted that it was difficult to analyze the results of various
panels. It was painfully obvious that the current comment based review
process does not scale to hundreds of proposals. Adding a numeric scale
to these metrics would help quantify the reviews. These numeric metrics
could then be used to provide a framework upon which the reviews,
proposal, panels, and programs can be compared. Possible scales for
these metrics would be:

-   Intellectual merit: excellent (4), very good (3), good (2), fair
    (1), poor (0)

-   Benefits: very large (4), large (3), medium(2), small (1),
    insignificant (0)

-   Creativity: very novel (4), novel (3), average (2), obvious (1),
    already done (0).

-   Time scale: 15+ years (4), 10 -- 15 years (3), 5 -- 10 years (2), 1
    -- 5 years (1), immediate (0)

-   Breadth of Applicability: very broad (society) (4), broad
    (scientific or technical) (3), average (academic discipline) (2),
    niche (1), special case (0)

-   Risk: very risky (4), risky(3), aggressive (2), incremental (1),
    obvious (0)

-   Cost: bargain (4), fair (3), pricey (2), inflated (1), outrageous
    (0)

-   Familiarity of the reviewer with the topic: very familiar (4),
    familiar (3), general knowledge (2), little knowledge (1), no
    knowledge (0)

These metrics are similar to those used by various journals and should
not be a burden on the reviewers.

This data could be used to construct a simple review database in which
each REVIEW record consisted of:

> REVIEW record: Panel ID, reviewer ID, proposal ID, budget, date,
> overall evaluation, Intellectual merit, benefits, novelty, timescale,
> breadth, risk, cost, familiarity.

This database could be used by the panel to compare the proposals. A
weighted average of the numeric metrics could be used to provide an
initial ranking of the proposals.

This database could be combined with the final proposal ranking and
final funding decision to construct a simple proposal database in which
each PROPOSAL record consisted of:

> PROPOSAL record: Panel ID, proposal ID, date, max overall evaluation,
> min overall evaluation, average of intellectual merit, average of
> benefits, average of novelty, average of timescale, average of
> breadth, average of risk, average of cost, average of familiarity,
> weighted ranking, actual ranking, funding decision

The NSF could use the REVIEW database to monitor the balance of
intellectual merit, benefits, novelty, timescale, breadth, risk, and
cost of the submitted proposals. The COV could use the PROPOSAL database
to evaluate the performance of the various panels. The COV and NSF could
use the PROPOSAL database to evaluate the balance of the intellectual
merit, benefits, novelty, timescale, breadth, risk, and cost between the
funded and non-funded proposals.

It was also noticed that it was very difficult to "see" the bigger
picture. The availability of these databases in a spreadsheet compatible
form would enable the data to be plotted. Multi-dimensional "radar
charts" could be used to compare the numeric metrics of various
proposals. Scatter plots, pie charts, and histograms could be used to
show various distributions such as ranking versus budget, ranking versus
risk, etc. Error bars or "bubbles" could be used to represent the
variation in the various reviews. In perspective, the use of numeric
metrics enables a whole range of analytic tools to be used. The use of
this data and these tools to augment the human element of the selection
process should have a great impact on the integrity and efficiency of
the program\'s processes and management.

###### 2.6 Network Connections

HPNC is a \$2M/year program within ANIR whose goal is to enable research
and education at the forefront of science and engineering via the
establishment of high performance (45M bits per second or greater)
internet connections to a national research network; prepare the next
generation of scientists, engineers, and other researchers, especially
individuals in traditionally underserved groups, to use advanced
networking in support of discovery, learning and innovation. The HPNC
program experienced a major transition during the FY2000-2002 review
period. During the early years of the program, \$300,000 awards were
made to major universities for high speed research network connections.
Proposals were driven primarily by the growing needs of the academic
research community for increased internet bandwidth to access
distributed computational resources and data services.

In FY2001, the program shifted to encourage proposals from museums,
research laboratories, minority serving institutions, historically Black
colleges and universities and non-profits. Award sizes also decreased
during this time from \$300,000 to \$150,000. The HPNC, at this point,
began to transition to more of an outreach and mentoring program. This
transition, coupled with the overall growing number of proposals coming
into the division, increased the workload of the program managers.
Institutions proposing to the HPNC program from the extended community
had less technical expertise in networking than previous institutions
funded under the program and they were also less experienced in proposal
writing. Program managers got more involved in working with the PIs to
enhance their proposals and enlisted long-time ANIR grantee, the
National Laboratory for Advanced Networking Research (NLANR), to assist
in details of technical implementation. Regional workshops were held to
assist with these efforts.

One of the effects of this program transition was that the quality of
pre-award documentation also declined. Based on a random sampling of
award jackets, responses to issues raised by review panels were often
not documented. Documentation in some jackets jumped from the panel
recommendation not to fund a proposal to the Form 7 recommending a
funding award with no explanation for the transition. During the CoV
meeting, the program manager was able to produce documentation that was
not in the award jacket leading the Committee to conclude that the
documentation oversights were caused by program manager overload. The
COV believes that the quality of documentation observed in a sampling of
HPNC award folders would put the division at risk during an audit. ANIR
must improve the quality of documentation in its award jackets. It is
noted that the COV has recommended elsewhere in this report that ANIR
needs additional staffing, which it believes will allow for improved
pre- and post-award administration of ANIR's grant and cooperative
agreement portfolio.

The above comments notwithstanding, there is evidence that the HPNC
program as it has evolved is successful in achieving its outreach goals.
The CoV observed evidence that the mentoring provided by both the NSF
and NLANR personnel has helped advance the program toward the
Foundation's goal of growing the technical and scientific workforce in
the US. Proposals declined in FY2000 were resubmitted and were funded in
FY2001 and applied for and received funding for other ANIR programs (REU
and STI) in FY2002. In addition, PIs on some of these proposals joined
the reviewer pool during the same period of time.

The Committee looked briefly at the International Connections program.
Overall, the program is performing a useful service: encouraging
high-performance connectivity to locations outside the US both in
support of networking research and NSF science more generally.

The Committee had one concern, which is that it could discern no guiding
principles for determining which international connections were of
greatest value. The current program is heavily focused on subsidizing
high-performance connectivity to long-standing research partners in
first-world countries. This preference is partially justified by the
fact that the first-world partners are prepared to subsidize part of the
cost. Although this is a reasonable policy, equally valid arguments
could be made in favor of funding links in support of NSF science in
Antarctica, and parts of South and Central America, and from the
perspective of enhancing US competitiveness, it might be better to
invest money in connecting regions that provide the best-educated
immigrants (which tend not to be first world countries). The Committee
was concerned that there were few signs that the choices had been
weighed. (Expressed another way: the Committee felt that ANIR may well
have made the correct funding decisions but in the absence of some
guiding principles it was hard to tell.)

ANIR/CISE might also consider working with agencies such as USAID and
the World Bank to investigate developing connections and supporting
infrastructure for developing countries.

*The Committee recommends that ANIR develop a set of priorities or
guiding principles for how International Connections monies are to be
spent. Simply holding a workshop where interested parties could discuss
the tradeoffs and make some recommendations would likely strengthen the
program and substantially clarify its mission.*

###### 2.7 Collaboration Between Networking and Other Disciplines

**2.7.1 Technology Transfer\
**\
It would benefit both CISE and NSF to better understand how technology
transfer is achieved. There are a number CISE programs, both current
(middleware) and contemplated (wireless), where technology transfer
plays a key role.

Technology transfer is always difficult.  There are a great many
problems in going from technology that works in a research environment
to technology that is a working product that consumers believe is
beneficial and worth purchasing.  A large set of issues determine how
successful any given technology transfer will be.  Such issues include
sociological, political, psychological, business management, and
economic issues, as well as the fundamental technology that is the focus
of the original research.

There's also a problem of defining technology transfer. Many people
define technology transfer as simply getting the relevant technology
incorporated into a product that is available for purchase. However, in
the recent technology bubble, excess venture capital created a
marketplace with lots of product available for purchase that no one
wanted to buy. We may need a better definition that encompasses the
notion that the technology is transferred in a way that provides benefit
to the technology's ultimate user or consumer. And we probably need
better technology transfer strategies that do not assume a high
availability of venture capital.

It is clear that a principled study of the factors that enable effective
technology transfer would be tremendously useful to NSF, CISE, ANIR and,
indeed, the nation at large. We would encourage NSF to take the steps
necessary to produce a study (which is, by necessity multi-discipline
and multi-dimension and thus logically, Foundation-wide).

However, as an interim step, we strong recommend that CISE and ANIR
spend a little time thinking about their own technology transfer goals
and methods for achieving those goals. Even something as simple as one
or two workshops in which researchers counsel CISE and each other about
methods of technology transfer, their relative merits, and benefits,
would be a strong contribution and likely to give valuable guidance to
CISE/ANIR programs where technology transfer is a goal.

###### 2.8 Post-Award Management of Projects

It was noticed that the management of the cooperative agreement awards
was spotty. It was also noticed that the annual reports also varied
widely in quality. It would help to get the community more involved. It
was noted that the scientific community is self-policing. Publications
are reviewed, published, and debated. It would be difficult for the NSF
to internally duplicate the integrity of this process. The NSF should
consider moving towards a "sunshine" policy in which the awarded
proposals and annual reports were made available on the web. This would
speed and facilitate the distribution of results, increase the potential
for collaboration, and enhance community involvement. While there might
be intellectual property and privacy issues involved, a step in this
direction would be to make the posting of the winning proposals and
annual reports on the web an option of the awardees.

**2.8.1 Post-Panel Process\
**\
A significant percentage of proposals recommended for funding are
recommended with specific conditions.  Often these conditions involve
the PIs' preparing addenda to the proposals, or providing other
information to verify that conditions have been met.  Such addenda and
additional information need to be subject to the same degree of rigorous
review for content as are the proposals themselves.  We realize that
there are processes in place to ensure that PIs meet the syntactic
requirement of conditions, and to ensure that PIs are in contact with
knowledgeable people to help with conditions that may be out of the PIs'
area of expertise.  However, we believe that ANIR should institute a
policy ensuring semantic review of these post-panel documents.  For
example, the reviewers who read the proposal and wrote reviews, or those
who suggested the conditions, might be asked to review the additional
material.\
\
**2.8.2 Post-Award Process\
**\
Unlike grants, cooperative agreements require significant technical
management, to ensure that contractual directions are carried out
effectively.  Such management includes (among other things) ensuring
that directives made in an award but not contained in the original
proposal, are carried out effectively.  Also, as with all research,
things may not go as planned.  Unlike the case in grant management, it
is the responsibility of the ANIR Program Director, and not only the PI,
to consider alternative courses of action and correction.  Often with
cooperative agreements, there is additional responsibility for PI\'s to
cooperate with other PIs.  It is very important to provide oversight and
direction for such interactions.  Finally, there is a responsibility to
provide constructive critique and offer useful suggestions, where
possible.\
\
It is the Committee's judgment that insufficient effort and attention
have been given to the management of cooperative agreements.  While
recognizing that Program Directors are heavily overloaded, and that, as
a result, performance must slip somewhere, the CoV believes that
management of cooperative agreements is not a good place for performance
to slip.  One positive suggestion the COV can make is to let more
"daylight" fall on projects, by publishing funded proposals and annual
and final reports prominently on the web.  Multiple eyes can often help
catch bugs, and publication may motivate better written reports.  We are
convinced that any intellectual property issues arising from such
publication can be dealt with by providing appropriate policies.

###### 2.9 Responses to the Last CoV

Specific concerns of the previous CoV (FY98-00) and ANIR's responses to
these concerns are discussed next.

1.  *The Division should provide relief from the heavy workloads on
    Program Officers and staff*. The Division increased the number of
    Program Officers, providing some relief. Counteracting these efforts
    was the growth in the number of proposals submitted and processed
    during the current reporting period. The ranks of staff did not
    appear to enjoy the same growth rate as did the ranks of Program
    Officers. Thus, it would appear that the Division's personnel are
    still beset with the unreasonable workloads that existed at the end
    of the last reporting period.

2.  *The Division should cease funding backbone network facilities, with
    the exception of international connections.* The Division now funds
    only international backbone infrastructure. This transition has been
    accomplished without negative impact on the level of service
    delivered to high-bandwidth science users.

3.  *The Division should pursue a program of developing* *infrastructure
    for a broad spectrum of applications.* The Division's focus on
    applying network technologies to new application areas was very
    apparent in the current reporting period. Most of the emphasis has
    been on opening up new areas of e-science. While collaboration
    between network researchers and scientists from other domain areas
    is seen as a natural fit, it was noted by the CoV that more emphasis
    could have been placed on fostering a "wide spectrum of application
    areas, including but not limited to the sciences, business,
    commerce, and society in general (e.g. job training, health care,
    and the political process)", to quote from the previous CoV report.
    The sentiment that ANIR's programs (research specifically, but also
    infrastructure) should not revolve exclusively around the foci of a
    small core of physical sciences was obvious in the CoV. Asking that
    computer science serves as the handmaiden of physical sciences is
    not the way to pursue good science, as experience has shown that the
    most-compelling ideas are those possessing an underlying generality
    that gives utility across a wide range of application areas. This is
    not to say that networking people should not collaborate with
    physicists or chemists or biologists -- to the contrary these have
    been some of the most fruitful marriages. It is, however, beneficial
    to give networking researchers greater leeway in selecting the
    application domains in which to apply their interests. From the
    historical perspective, in the early 1990's networking research was
    being pushed down a path to pursue applications epitomized by the
    Gigabit Testbeds and the vBNS, but the resistance of most members of
    the community to go down this path turned out to be the correct
    instinct. What resulted was a modern Internet, better attuned to the
    requirements of a general public than to specific needs of a small
    segment of power users. The utility of general services was
    universal and applies equally well across the broader spectrum of
    all users.

4.  *The Division should support infrastructure for use by networking
    researchers.* The use of specialized infrastructure by network
    researchers was deemed to be an essential element of a healthy,
    experimentally-oriented community. Conventional infrastructure
    cannot be easily shared by production users and researchers engaged
    in the development and testing of experimental protocols and
    services, because experimental technologies can disrupt production
    users and even destabilize an entire network. Certainly, few users
    are willing to put up with such distractions. Much discussion has
    been dedicated to understanding what facilities should be provided
    and how resources should be managed in such infrastructure.
    Moreover, approaches about how to implement and operate a research
    testbed in a cost-effective manner have been presented and debated.
    Creative ideas for building flexible frameworks to experiment with
    new protocols and services as overlays atop infrastructure that is
    shared with productions users have been successfully analyzed and
    demonstrated. NSF has been squarely in the center of these
    discussions, having conducted a number of workshops dedicated
    entirely to the building and maintenance of experimental network
    testbeds. It was upon these foundations that NSF recently announced
    programs to build testbeds that could be used to evaluate routing
    protocols, denial-of-service attacks and defenses, peer-to-peer
    networks, and new link technologies. A healthy response to the
    announcements has been noted. Although the process has taken longer
    than expected, the CoV deemed NSF's careful examination of the
    issues to be prudent and responsible.

5.  *The Division should plan, implement, and actively manage a network
    middleware initiative with the goal of delivering key software
    components and core services.* The Division responded by introducing
    the NMI program early in the current reporting period. Taking into
    consideration the many recommendations and objectives provided by
    the previous CoV, ANIR developed a comprehensive program, awarded
    grants and cooperative-agreement projects, and initiated production
    of a well-integrated suite of middleware for use by the scientific
    community. One can say that NMI accomplished its major milestone of
    launching science-driven middleware into the broader market. The
    ascent of Grid technology and its widespread adoption by the
    computer industry are noteworthy indicators that NMI has made the
    desired impact. With the establishment of the Global Grid Forum --
    based on the successful Internet Engineering Task Force model -- the
    Grid has been guided to a stage comparable to where NSF brought the
    Internet in the previous decade.

6.  *The Division should increase the fraction of its budget committed
    to research activities.* The Division set for itself the goal of
    allocating at least 50% of its overall budget to research programs.
    At this moment more than 60% of its overall budget goes to research
    programs.

7.  *The Division should place greater emphasis on reporting GPRA
    outcomes, forcing PIs to be more responsive in reporting their
    accomplishments.* The CoV's sampling of ANIR's proposal jackets
    indicates that PIs' reporting of results and submissions of interim
    and final reports is no longer a significant problem. Aggressive and
    universal use of the Fastlane system has enabled effective tracking,
    monitoring, and signoff of deliverable reports. The Division tightly
    enforces adherence to the requirements that all proposals and
    reports address the issues of desired and achieved outcomes for in
    terms of the NSF merit-review criteria as well as in terms of
    people, ideas, and tools.

###### 

###### 3. Improving the Value of the CoV

The CoV recommends several ways in which the CoV process could be
improved.

The Committee found that the two days of meetings are filled with an
enormous amount of information gathering and processing. The NSF staff
should be commended for the herculean task of making the data available
to the CoV. However, there is too much data to properly synthesize in
the time allotted to the meetings themselves. To make better use of the
time that Committee members spend assembled in the same location, the
CoV process should include a preview stage, where a subcommittee of the

CoV (at least one cochair, and a total of three or four people) visits
NSF, reviews the materials planned to be provided to the participants,
and suggests additional materials to be provided. The actual CoV meeting
should occur between one and three weeks later.

The initial subcommittee should delegate specific tasks to individual
Committee members and should prepare a list of suggested questions to be
considered by the Committee. The subcommittee should indicate what
information will be available at the meeting, in order to give Committee
members a chance to prepare and to make additional recommendations about
data summarization.

It was tremendously helpful to have on-line access to CoV-related
reading materials in advance of the meeting as web links through a
secure site. This had a direct impact on the efficient use of time
during the on-site meeting. Therefore, it would be beneficial to the CoV
process to make the complete set of reading materials available on-line
and to make them accessible for a window of time both before and after
the meeting itself. In

addition, Committee members commented that the availability of on-line
materials would eliminate the need to make printed copies, saving NSF
staff an enormous amount of time and money, saving many a tree, and
saving CoV members from transporting such weighty documentation.

The previous CoV bemoaned the lack of statistical tabulations of data
that would have been useful in making their evaluations and
recommendations. The current CoV echoes this sentiment. The current CoV
did not receive all the statistical compilations suggested by the
previous CoV, found additional ways in which they preferred to view the
data, and can guess that future CoVs will want to compile the data in
yet other formats. Thus, it is critical to provide a more flexible means
of processing and displaying the data; it would greatly simplify the
evaluation process to have access to a flexible database that the CoV or
NSF staff could query and could use to produce statistical tabulations
as required. Such a setup would considerably improve the quality of
information available to the CoV. In addition to raw data and
statistical tabulations, the ability to generate graphically represented
data summaries would be very useful.

CoV members found it difficult to complete the CoV process as it
traditionally has been conducted. The process historically has centered
around the review of a sample of the Division\'s proposal jackets, which
is a painstaking effort that members felt gave only a partial picture of
the overall state of the Division\'s projects. Although this process
allows members to evaluate individual proposals, it makes it difficult
to answer questions about a specific panel or an entire program, since
this requires CoV members to wade through a large collection of related
jackets in the short amount of time allotted to the review.
Specifically, better panel-wide and program-wide information must be
made available to the CoV, and the Committee should have the ability to
ask for and receive this data (e.g., summary reports) in near real time.
Although the Division provided a large amount of data and reports to the
CoV and made valiant efforts to compile specifically requested reports
in a timely fashion, this process was viewed as less than optimal;
again, the CoV recommends the Division make available a database that is
populated with information stored in the individual proposal jackets,
but that could be queried to cut the data across a given panel or
program. Such a solution would make more efficient use of the time spent
by CoV members, as well as NSF staff.

###### 4. Answers to Core Questions

The Committee's response to the official NSF core questions follows. The
research, ITR, and infrastructure programs are discussed integrally for
each question. Comments that are specific to a program are identified as
being so.

###### 

+-----------------------------------------------------------------------+
| ##### Date of COV: May 8 & 9, 2003                                    |
+-----------------------------------------------------------------------+
| **Program/Cluster: Network Research, Information Technology Research, |
| Infrastructure**                                                      |
+-----------------------------------------------------------------------+
| **Division: ANIR**                                                    |
+-----------------------------------------------------------------------+
| **Directorate: CISE**                                                 |
+-----------------------------------------------------------------------+
| **Number of actions reviewed by COV**[^5]**: Awards: Declinations:    |
| Other:**                                                              |
+-----------------------------------------------------------------------+
| **Total number of actions within Program/Cluster/Division during      |
| period being reviewed by COV**[^6]**: Awards: Declinations: Other:**  |
+-----------------------------------------------------------------------+
| **Manner in which reviewed actions were selected: Random sampling**   |
+-----------------------------------------------------------------------+

**PART A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures. **Provide comments in the space below the
question**. **Discuss areas of concern in the space provided.**

+-----------------------------------------------------------+----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES**  | **YES,   |
|                                                           | NO,**    |
|                                                           |          |
|                                                           | **DATA   |
|                                                           | NOT      |
|                                                           | AV       |
|                                                           | AILABLE, |
|                                                           | or NOT   |
|                                                           | APPL     |
|                                                           | ICABLE** |
+-----------------------------------------------------------+----------+
| Is the review mechanism appropriate? (panels, ad hoc      | Yes      |
| reviews, site visits)                                     |          |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Is the review process efficient and effective?            | Yes      |
|                                                           |          |
| Comments: The difficulty of obtaining panelists without   |          |
| conflicts of interest (as currently defined) has made the |          |
| process less efficient.                                   |          |
+-----------------------------------------------------------+----------+
| Are reviews consistent with priorities and criteria       | Yes      |
| stated in the program's solicitations, announcements, and |          |
| guidelines?                                               |          |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Do the individual reviews (either mail or panel) provide  | Yes      |
| sufficient information for the principal investigator(s)  |          |
| to understand the basis for the reviewer's                |          |
| recommendation?                                           |          |
|                                                           |          |
| Comments: A number of the reviews were considered too     |          |
| brief or superficial to provide sufficient feedback to    |          |
| principal investigators.                                  |          |
+-----------------------------------------------------------+----------+
| Do the panel summaries provide sufficient information for | Yes      |
| the principal investigator(s) to understand the basis for |          |
| the panel recommendation?                                 |          |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Is the documentation for recommendations complete, and    | Yes      |
| does the program officer provide sufficient information   |          |
| and justification for her/his recommendation?             |          |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Is the time to decision appropriate?                      | Yes      |
|                                                           |          |
| Comments: Remarkably, the dwell time shrank despite       |          |
| substantial increase in the number of proposals.          |          |
+-----------------------------------------------------------+----------+
| Discuss issues identified by the COV concerning the       |          |
| quality and effectiveness of the program's use of merit   |          |
| review procedures: Increasing numbers of proposals have   |          |
| made it difficult to recruit qualified panelists. Not     |          |
| only are more panelists required to review proposals, but |          |
| many qualified panelists must be excused from service     |          |
| because their own proposal submissions disqualify them    |          |
| under conflict-of-interest rules.                         |          |
+-----------------------------------------------------------+----------+

A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers. **Provide comments in the space below the question.
Discuss issues or concerns in the space provided.**

+---------------------------------------------------------+------------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**         | **YES,     |
|                                                         | NO,**      |
|                                                         |            |
|                                                         | **DATA NOT |
|                                                         | AVAILABLE, |
|                                                         | or NOT     |
|                                                         | AP         |
|                                                         | PLICABLE** |
+---------------------------------------------------------+------------+
| Have the individual reviews (either mail or panel)      | Yes        |
| addressed whether the proposal contributes to both      |            |
| merit review criteria?                                  |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Have the panel summary reviews addressed whether the    | Yes        |
| proposal contributes to both merit review criteria?     |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Have the *review analyses* (Form 7s) addressed whether  | Yes        |
| the proposal contributes to both merit review criteria? |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Discuss any issues or concerns the CoV has identified   |            |
| with respect to NSF's merit review system.              |            |
+---------------------------------------------------------+------------+

A.3 Questions concerning the selection of reviewers. **Provide comments
in the space below the question. Discuss areas of concern in the space
provided.**

+---------------------------------------------------------+------------+
| Selection of Reviewers                                  | YES , NO,  |
|                                                         |            |
|                                                         | DATA NOT   |
|                                                         | AVAILABLE, |
|                                                         |            |
|                                                         | or NOT     |
|                                                         | APPLICABLE |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       | Yes        |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            | No         |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments: The CoV believes that there was considerable  |            |
| room for improvement in panel composition. Many of the  |            |
| problems would simply be resolved by setting panel      |            |
| dates and recruiting a subset of the panel membership   |            |
| well in advance of the proposal submission deadlines.   |            |
| The problem is covered in greater detail in the         |            |
| discussion of concerns below.                           |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    | Yes        |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments: The CoV examined the total set of panelists   |            |
| for all programs by year for 2000, 2001, and 2002. In   |            |
| 2000, the fraction of panelists from institutions       |            |
| physically close to NSF (from the area roughly bounded  |            |
| by Charlottesville, Richmond and Baltimore) was greater |            |
| than 9%, which appeared high to the CoV and may have    |            |
| reflected program officers relying too heavily on local |            |
| researchers, who are easier to schedule.                |            |
|                                                         |            |
| In 2001 and 2002 the Committee found no such obvious    |            |
| discrepancies.                                          |            |
|                                                         |            |
| 2000 was also exceptional in having a small number of   |            |
| panels. So,                                             |            |
|                                                         |            |
| it seems likely that 2000 was an anomaly. The next CoV  |            |
| should conduct a similar check to make sure.            |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      | Yes        |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
|                                                         |            |
| A subcommittee of the CoV reviewed the basic networking |            |
| research program. Overall, the subcommittee\'s view was |            |
| positive. ANIR is funding strong research, and its      |            |
| panels generally seem to be selecting the outstanding   |            |
| proposals from those submitted. We were also gratified  |            |
| and pleased to see that panels periodically sought out  |            |
| high-risk, high-reward work.                            |            |
|                                                         |            |
| The one major point of concern was the quality of the   |            |
| panels. While most panels have some stellar researchers |            |
| on them, it is also clear that any panels were of       |            |
| uneven quality, and that the panels were sometimes      |            |
| challenged to evaluate properly the proposals in front  |            |
| of them. Discussions with ANIR staff (present and past) |            |
| indicate that the problem of panel quality stems from   |            |
| two issues:                                             |            |
|                                                         |            |
| 1\. The best people are often the busiest people. In    |            |
| particular, to get top researchers on a panel requires  |            |
| that a panel date be set months in advance.             |            |
| Historically, ANIR has been unwilling to set a panel    |            |
| date and to select panel personnel until it has         |            |
| proposals in hand. The theory is that the actual        |            |
| proposals received determine the kind of panel          |            |
| expertise required.                                     |            |
|                                                         |            |
| 2\. NSF conflict-of-interest rules can unnecessarily    |            |
| hinder the selection of good panelists. In new \"hot\"  |            |
| research areas, it is likely that all the researchers   |            |
| in the area will submit proposals. Under NSF            |            |
| conflict-of-interest rules, this disqualifies all the   |            |
| researchers from reviewing. This result undermines the  |            |
| peer review process -- as one panelist noted, "your     |            |
| best critic is usually your closest competitor".        |            |
|                                                         |            |
| The committee recommends ANIR take two steps to address |            |
| the issue of panel quality:                             |            |
|                                                         |            |
| 1\. Set panel dates far in advance and invite some      |            |
| panelists before the proposals are submitted. Then once |            |
| the proposals are received, invite additional people to |            |
| join the panel to provide the additional technical      |            |
| expertise needed.                                       |            |
|                                                         |            |
| 2\. Find a way (more aggressive use of                  |            |
| conflict-of-interest waivers was suggested) to get      |            |
| competitors\' inputs on proposals in narrow fields      |            |
| where all the senior researchers have submitted         |            |
| proposals.                                              |            |
|                                                         |            |
| Other ideas discussed included using standing panels to |            |
| review proposals. Such arrangements, which are used in  |            |
| NIH and other NSF directorates, make selection of       |            |
| panelists and scheduling of panel meetings less         |            |
| problematic. A known fixed term also gives the panelist |            |
| some idea of how much effort they will be expected to   |            |
| invest. As practiced currently, requests for panel      |            |
| service are often unpredictable and one never knows how |            |
| frequently one will be asked to serve.                  |            |
|                                                         |            |
| A lesser point of concern was the appearance of         |            |
| \"funding fads\". For instance, in one panel            |            |
| approximately two-thirds of the awards were in          |            |
| mobile/ad-hoc/wireless networks. While it is certainly  |            |
| true that topics become hot, we would be stunned if the |            |
| choice of a particular topic led to ideas being so much |            |
| better than the norm that they merited two-thirds of    |            |
| the awards. We encourage ANIR to be more alert to these |            |
| kinds of fads and to encourage panelists to look deeper |            |
| at proposal quality rather than timeliness or current   |            |
| excitement.                                             |            |
+---------------------------------------------------------+------------+

A.4 Questions concerning the resulting portfolio of awards under
review**. Provide comments in the space below the question. Discuss
areas of concern in the space provided.**

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     | Appropriate   |
| projects supported by the program.                   |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  | Appropriate   |
| scope of the projects?                               |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   High Risk Proposals?                             |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Multidisciplinary Proposals?                     |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Innovative Proposals?                            |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Funding for centers, groups and awards to        |               |
|     individuals?                                     |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Awards to new investigators?                     |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Inappropriate |
| balance of:                                          |               |
|                                                      |               |
| -   Geographical distribution of Principal           |               |
|     Investigators?                                   |               |
|                                                      |               |
| Comments: Overall, the ANIR portfolio appears to be  |               |
| fairly balanced with respect to the geographical     |               |
| distribution of PIs. It was noted, however, that the |               |
| NMI program had an abnormally large number of awards |               |
| based in Illinois. Of 25 awards, eight are to        |               |
| Illinois institutions. This proportion of nearly     |               |
| one-third seems to be incompatible with the desire   |               |
| for geographical balance.                            |               |
|                                                      |               |
| Out of the NMI base of 25 awards, two states         |               |
| (Illinois and California) received 12 awards. It     |               |
| appears inappropriate that about half of all NMI     |               |
| awards have gone to Illinois or California.          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Institutional types?                             |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Projects that integrate research and education?  |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance:                                             |               |
|                                                      |               |
| -   Across disciplines and subdisciplines of the     |               |
|     activity and of emerging opportunities?          |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          | Appropriate   |
| participation of underrepresented groups?            |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      | Appropriate   |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
|                                                      |               |
| The Funding Rate in percent for Minority (55%, 25%,  |               |
| 22%), Minority Involvement (41%, 52%, 22%), Gender   |               |
| -- Female (57%, 33%, 22%), and Women Involvement     |               |
| (49%, 41%, 23%) fell over the period (2000, 2001,    |               |
| 2002); however, the average funding rate for all     |               |
| categories also fell for this period (31%, 39%,      |               |
| 21%). The annual funding rate for all of these       |               |
| categories except for Minority (2001) was above      |               |
| average.                                             |               |
+------------------------------------------------------+---------------+

![](media/image2.wmf){width="5.895833333333333in" height="3.21875in"}

A.5 Management of the program under review**. Please comment on:**

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments: The post-award management of cooperative agreements, as     |
| reflected by the jackets, was seen to have gaps that hampered the     |
| CoV's understanding of some of the programs.                          |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments: The programs were very flexible, and the year-to-year       |
| changes evident in them indicate the Division's willingness to        |
| respond and adapt to new trends in the field, in terms of research,   |
| education, and infrastructure development and deployment.             |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments: The Division has implemented the core of recommendations    |
| made by the previous CoV with respect to prioritization of the        |
| portfolio. The Division has also shown sound planning in aligning its |
| supported research with the overarching goals of the ITR program.     |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
+-----------------------------------------------------------------------+

##### PART B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.**

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

+-----------------------------------------------------------------------+
| **B.1 [NSF OUTCOME GOAL for PEOPLE]{.underline}: Developing "a        |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| ANIR has taken steps to foster a better workforce in a number of      |
| ways. Particularly important has been the direct support offered to   |
| attendees of key conferences. Over the reporting period, the Network  |
| Research Program has offered travel grants for researchers and        |
| students to attend IEEE INFOCOM 2001, 2002, and 2003; ACM Sigcomm     |
| 2002; and ACM Mobicom 2002. These are counted as three of the most    |
| important conferences in networking. The Division also sponsored      |
| timely workshops to examine the establishment and use of network      |
| research testbeds and the issues of modeling the Internet as a        |
| large-scale complex system.                                           |
|                                                                       |
| The Network Infrastructure Program has supported workshops to address |
| the integration of production, advanced development, and research     |
| testbeds. Outreach to members of the NMI Program included meetings    |
| and an extensive project website to provide information for potential |
| adopters and application developers.                                  |
|                                                                       |
| The Division also organized PI meetings that brought together the     |
| members of the community to present and discuss their work. The       |
| meetings have included a series of panel discussions that cover many  |
| of the field's burning issues.                                        |
+-----------------------------------------------------------------------+
| **B.2 [NSF OUTCOME GOAL for IDEAS]{.underline}: Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The Division has done very well in enabling discovery across the      |
| frontiers of science and engineering. ANIR's balanced approach of     |
| funding fundamental research on the processes and principles of       |
| networks, systems research to demonstrate and develop new ideas in    |
| large-scale software and services for networks, and leading-edge      |
| testing and deployment of new networking technologies.                |
|                                                                       |
| N. McKeown of Stanford has employed ANIR funding to develop new ideas |
| to allow core routers to keep pace with the unrelenting growth in     |
| traffic demands that these devices are required to handle. The highly |
| parallel Fork-Join router is being designed to support Tb/s aggregate |
| throughput by using special emulation techniques that allow the       |
| internal switching fabric to achieve the superior performance of an   |
| output-queued switch without the associated drawbacks that output     |
| queueing introduces. New ideas for redesigning routers to handle      |
| streaming data rather than traditional web traffic have pointed to    |
| fruitful techniques for future generations of routers. This work has  |
| enriched the technology base of switch and router design, and has     |
| been incorporated into the PI's startup company, Abrizio.             |
|                                                                       |
| Using survey techniques from the social sciences, J. Cole of UCLA has |
| developed a longitudinal study of the impact of networked PCs on US   |
| society. The survey has quantified over time several characteristics  |
| of network usage by individuals and households. The work provides     |
| much-needed scientific insight into the changing patterns of Internet |
| use and resource consumption.                                         |
|                                                                       |
| J. Feigenbaum of Yale developed incentive-compatible protocols for    |
| interdomain routing to replace protocols based on bilateral           |
| contracts. This approach can more effectively optimize system-wide    |
| problems, such as globally efficient routing and the resolution of    |
| conflicting policy requirements.                                      |
|                                                                       |
| To serve medical practitioners with mobile, wireless, handheld        |
| devices, R. Bagrodia of UCLA developed the iMASH system for           |
| performing session-level handoff in a way effectively favors the      |
| sharing and secure transmission of multimedia streaming data over a   |
| bandwidth-limited and intermittently connected wireless network.      |
|                                                                       |
| Motivated by the unexpected appearance of flash crowds in networks,   |
| such as were experienced during the September 11 crisis, D.           |
| Rubenstein of Columbia designed protocols and supporting mechanisms   |
| to mitigate temporary, unpredictable oversubscription of network      |
| resources. The approaches include better techniques for predicting    |
| flash crowds from trace data and pooling together resources to meet   |
| swamping demands, e.g. by clever caching of highly sought-after hot   |
| content.                                                              |
+-----------------------------------------------------------------------+
| **B.3 [OUTCOME GOAL for TOOLS:]{.underline} Providing "broadly        |
| accessible, state-of-the-art and shared research and education        |
| tools."**                                                             |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The Division may pride itself on having produced tools that are       |
| extremely valuable to the research community. Three examples of       |
| immense importance emerged during the Committee's review: NLANR/MOAT  |
| datasets, ns-2 protocol simulation software, and the Emulab network   |
| testbed.                                                              |
|                                                                       |
| Based at UCSD's SDSC, the NLANR project under H. Braun and K. Claffy  |
| has collected a large set of historical packet traces from a variety  |
| of monitoring points in the Internet. These datasets are used to      |
| drive high-fidelity simulations, to develop traffic models, and to    |
| mine for the discovery of trends and patterns that can suggest        |
| improvements in network design and operation. A high proportion of    |
| the community's archival publications that deal with empirical data   |
| have relied on these traces.                                          |
|                                                                       |
| The ns-2 simulation package is probably the one most preferred and    |
| employed by researchers who design end-to-end and routing protocols.  |
| Under continual development at USC/ISI as part of the CONSER project  |
| led by J. Heidemann, ns-2 has been applied extensively to the design  |
| of congestion control in TCP, analysis of multicast routing, and      |
| evaluation of techniques to support multimedia and streaming          |
| protocols. Again, many publications on protocol design could not have |
| been completed without ns-2. As a vital link between education and    |
| research, ANIR also supported workshops, tutorials, and a mailing     |
| list for ns-2 users, emphasizing the introduction of the tool to      |
| graduate students who are embarking on their research careers.        |
|                                                                       |
| The Emulab facility -- based primarily at Utah under J. Lepreau's     |
| leadership but with outreach nodes at other institutions -- is a      |
| collection of cluster computers at several sites that can be used to  |
| emulate a network of dispersed nodes. Emulab supports a range of      |
| experimental modes and offers tools to general realistic scenarios.   |
| Again, the research community has enthusiastically embraced this      |
| tool, and the evidence of its importance may be seen in the sheer     |
| numbers of research papers that report results obtained through       |
| Emulab.                                                               |
|                                                                       |
| These three, broad-use tools underscore the favorable return on       |
| investment that the Division can expect from its tool-development     |
| agenda.                                                               |
+-----------------------------------------------------------------------+

##### PART C. OTHER TOPICS

C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.

C.2 Please provide comments as appropriate on the program's performance
in meeting program-specific goals and objectives that are not covered by
the above questions.

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

The CoV believes that the staffing and workload issues brought about by
rapid growth in the Division's programs (and discussed in detail
elsewhere in this report) remains as one of the most critical areas of
impact on the program's performance. Given the criticality and magnitude
of the problem, Foundation-wide measures to deal with the problem should
be considered.

C.4 Please provide comments on any other issues the COV feels are
relevant.

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the ANIR CoV

Joseph A. Bannister

Chair

######  Appendix A

###### Committee Methodology

The CoV members were chosen and solicited by NSF staff and the CoV
co-chairs. The CoV met at NSF headquarters for two days on May 8 and 9,
2003. An agenda prepared prior to the meeting by Dr. Tom Greene of NSF
and the co-chairs, is attached in Appendix B. Extensive briefings were
not given, but voluminous written materials were provided and NSF staff
were available at almost all times to answer questions and participate
in discussions.

For purposes of answering the core questions, the CoV was divided into
three subcommittees: research (five members), ITR (four members), and
infrastructure (four members). These groups questioned program managers,
examined written materials, and examined a number of proposal jackets in
detail. For the GPRA questions, the written materials provided by NSF
(statistics, lists of awards, and written interim and final reports)
were a primary resource.

For purposes of discussing issues surrounding the future of ANIR, the
CoV met in plenary session on the second morning, and in breakouts (to
develop different perspectives), and then in plenary (to resolve
differences and reach consensus positions). The plenary meeting
developed an outline of the report and assigned writing tasks to each
member of the CoV.

The meeting concluded with a meeting of the CoV with the Acting Director
and all but one of the Division staff to provide and discuss the
Committee's feedback. This session was brief, as the NSF staff
participated through most of the two days and were thus generally aware
of the Committee's conclusions.

**Appendix B**

**Working Agenda**

**Advanced Networking Infrastructure and Research (ANIR) Committee of
Visitors (COV)**

**May 8 & 9, 2003**

**Purpose: Assess the integrity and efficiency of the program's
processes and management**

**Thursday, May 8, 2003**

> 8:00 AM Coffee and pastries
>
> 8:30 AM Greetings from Deborah Crawford, Deputy Assistant Director,
> CISE
>
> 8:45 AM COV administration and introductions
>
> 9:00 AM Charge to the COV -- Mari Maeda, Acting Division Director,
> ANIR
>
> 9:20 AM Networking Research Programs Overview -- Taieb Znati, Senior
> Program Director, ANR
>
> 9:35 AM Infrastructure Programs Overview -- Tom Greene, Senior Program
> Director, ANI
>
> 9:50 AM Information Technology Research (ITR) Overview -- Taieb Znati,
> Senior Program Director, ANR

10:00 AM How to Read a Proposal/Award Jacket -- Gwen Hardenbergh

10:15 AM Guidance and instructions to COV -- Joseph Bannister and Ron
Hutchins

10:30 AM Breakout into three sub-panels (Research, Infrastructure, and
ITR)

-   Review program materials, statistics, and jackets for a random
    selection of proposals (one NSF staff member present to answer
    questions and provide guidance for each sub-panel)

12:00PM Lunch

> 1:00 PM Breakout (continued)
>
> 2:00 PM Sub-panel chairs consultation session

4:00 PM Plenary

-   Report of sub-panels and further discussion

5:00 PM Planning for Day 2

-   Make evening assignments in preparation for Day 2

-   Committee members read annual and final reports

6:30 PM Dinner (optional: group reservation will be made)

**Friday, May 9, 200**

8:00 AM Coffee and pastries

8:30 AM Plenary

8:45 AM Breakout

> • Assess the results (outputs and outcomes) of the program's
> investments

• Read selected project reports

• Form list of promising outcomes and discuss their quality and impact

12:00PM Lunch

1:00 PM Breakout (continued)

• Discuss general quality and impact of programs and suggestions for
improvement

• Topics to emphasize for the future

2:15 PM Plenary -- Discuss findings from breakout

• Report of sub-panels and further discussion

• Discussion of relationship of three program areas and other NSF
programs

3:00 PM Plenary -- Report Preparation

• Refine preliminary outline of report prepared by Chair

• Discuss responsibilities, coordination, and schedule for writing

3:30 PM Breakout

• Refine outline further

• List major points of each subsection

• Agree on writing assignments (attach name to each subsection)

> 4:00 PM COV and NSF staff

• Final questions to NSF staff

• Discussion of conclusions with NSF staff, clarification discussions

• Opportunity for NSF staff to respond to conclusions

5:00 PM Adjourn

# 

[^1]: David A. Patterson, *et al.*, *Looking over the Fence at
    Networks:* *A Neighbor's View of Networking Research,* National
    Academy of Sciences Press, Washington, 2001.

    (http://www7.nationalacademies.org/cstb/pub_lookingover.html)

[^2]: John A. White, Jr. *et al*., *Science and Engineering
    Infrastructure for the 21^st^* *Century:* *The Role of the National
    Science Foundation*, National Science Board Report NSB 02-190, April
    2003.

    (http://www.nsf.gov/nsb/documents/2003/start.htm)

[^3]: P. Freeman and W. Aspray, *The supply of information technology
    workers in the United States*, Computing Research Association,
    Washington, D.C., 1999.

[^4]: D. Atkins *et al*., *Revolutionizing Science and Engineering
    Through Cyberinfrastructure: Report of the National Science
    Foundation, Blue Ribbon Advisory Panel on Cyberinfrastructure*,
    January 2003. (<http://www.communitytechnology.org/nsf_ci_report/>)

[^5]: To be provided by NSF staff.

[^6]: To be provided by NSF staff.