-----------------------------------------------------------------------
  2003 Committee of Visitors

  -----------------------------------------------------------------------

Review of the Division of Computer and Communications Research of the
National Science Foundation

For the Reporting Period FY 2000 - 2003

  --------- -------------------------------------------------------------
  Section   

  1         
  --------- -------------------------------------------------------------

Committee Report

In compliance with the three-year review requirement, the National
Science Foundation's Division for Computer-Communications Research (CCR)
convened a Committee of Visitors (COV) for two days, June 5 and 6, 2003,
at NSF headquarters, Arlington, VA. The Committee was composed of
selected senior researchers in relevant specialty fields and was chosen
to span the scientific purview of the CCR Division. At the outset, the
Committee was provided comprehensive documentation for its two-day
meeting, including succinct descriptions of the charge to the Committee
and its obligations in responding. The meeting opened with welcoming
remarks by CISE Deputy Assistant Director Deborah Crawford and CCR
Division Director Kamal Abdali. Deputy Division Director Frank Anger
gave an overview of COV procedures. Overview presentations on the
activities of CCR over the review interval were then provided by and
Carmen Whitson. Committee Chair Richard DeMillo then gave an overview of
committee procedures and provided an outline for the final report (See
Appendix 5). A copy of the work agenda is given in Appendix 1. The
membership of the committee is reproduced in Appendix 2.

# Executive Summary and Recommendations

## Overview of CCR

The *Division of Computer-Communication Research (CCR)* supports
research in a broad array of disciplines ranging over theory, analysis,
design, construction, and utilization of computing and networking
hardware and software. Funding programs in the division are structured
in ten program units corresponding to major subject areas in computer
science and engineering.

-   Theory of Computing (TOC)

-   Design Automation for Micro and Nano-Systems (DA)

-   Graphics, Symbolic and Geometric Computation (GSG)

-   Communications Research (COM)

-   Computer System Architecture (CSA)

-   Signal Processing Systems (SPS)

-   Distributed Systems and Compilers (DSC)

-   Embedded and Hybrid Systems (EHS)

-   Software Engineering and Languages (SEL)

-   Trusted Computing (TC)

In addition, CCR supports research on algorithms and computing and
communication techniques to exploit emerging technologies such as
nanoscale and quantum devices. CCR also encourage multi-disciplinary
research in the context of computer science and engineering, including
research on algorithms and software for challenging computational
problems in science and engineering. This research is often supported
through the following special initiatives:

-   Highly Dependable Computing and Communication System Research
    (HDCCSR)

-   Computational and Algorithmic Representations of Geometric Objects
    (CARGO)

-   Sensors and Sensor Networks

-   Information Technology Research

## Trends and Statistics

The Division budget for research grants in FY02 was \$115.806M. This
represents 22.29% of the CISE research budget, an increase of 20.3% in a
year in which the overall CISE budget grew by 8.4%. The CCR increase was
apportioned among core and special programs as follows:

  -------------------------------- ------------ ------------ ------------
  Program                          FY01 (\$M)   FY02 (\$M)   \% Change

  Core Including Panel Expenses    59.8         60.4         1.0

  ITR                              34.6         46.2         33.3

  STC                              0            4.1          \--

  Nanotechnology                   1.8          5.2          180

  Total                            96.3         115.8        20.3
  -------------------------------- ------------ ------------ ------------

Figure 1 below shows the distribution of funds in the five divisions of
CISE and the office of the Assistant Director (A/D)

![](media/image1.wmf){width="3.7395833333333335in"
height="2.0416666666666665in"}

Figure 1: Distribution of CISE Research Funds.

CCR acted on 1281 proposals in 13 regular and special programs in FY02,
ultimately awarding 325 grants. This is an increase of 16% over the 1101
proposals acted on in FY01. Figure 2 shows the three-year trend in
submitted proposals for CCR. The 325 awards represent a success rate of
25%, a decline over the previous fiscal year. Figure 3 shows historical
success rates for CCR.

![](media/image2.wmf){width="6.351388888888889in"
height="3.6069444444444443in"}

Figure 2. Proposal Trends

![](media/image3.wmf){width="5.544444444444444in" height="3.15in"}

Figure 3. Historical Success rates

In making these awards, CCR called upon 867 members of the computer
science research community as reviewers. Figure 4 shows the historical
increase in the number of reviewers used by CCR. The overwhelming
majority of CCR awards are made by panels as opposed to individual or
*ad hoc* mail reviews. Most of the growth in panelists is due in part to
the increases in ITR submissions. However, as shown in Figure 5, there
has been substantial growth in panels due to regular program
submissions.

![](media/image4.wmf){width="5.1625in" height="2.932638888888889in"}

Figure 4. Total CCR Reviewers

![](media/image5.wmf){width="5.120138888888889in"
height="2.9034722222222222in"}

Figure 5. CCR Panelists

## Consensus Conclusions

NSF has organized and systemized its COV procedures (see Appendix 3),
and provided a FY2000 Report Template (Appendix 4) for consistent
evaluation across all programs. In direct compliance with the charge,
this template addresses integrity and efficiency of the programs,
processes and management of the Division, and the quality of scientific
results produced from NSF investments. This quality evaluation is
directly linked to satisfying the Government Performance and Results Act
of 1993 (GPRA).

Each sub-committee was instructed to respond to the template items with
the intent of revealing those points where consensus can be achieved
across the whole committee. The agenda of the two-day meeting was
designed so that periodic plenary sessions could coalesce topics from
the sub-committees for group analyses, compare status of the respective
studies, and identify common issues.

### Integrity and Efficiency Program Processes and Management

The committee examined all aspects of program operations and decisions,
using information supplied by NSF staff and information inferred from
direct examination of proposal jackets and decision documents. The
intent of examining program operations was to assess both the quality
and integrity of CCR operations. By the same token the examination of
program decisions was aimed at assessing technical and management
aspects of the actions taken by programs during the review period.

### Quality and Effectiveness of Merit Review Procedures

The Committee expressed overall satisfaction with both the quality and
effectiveness of the merit review process. In general, CCR program
managers carry out their duties with skill and professionalism, often
under conditions of considerable stress due to expanding workload and
limited research funding. The committee further determined that the CCR
community participates actively in the merit review processes of the
Foundation. Program managers assemble high-quality panels and reviewers
who are experts in their fields and who are representative of the
research community as a whole.

The Committee did come to several specific consensus conclusions in a
number of areas that have significant impact on CCR programs. In many
cases, the Committee made concrete recommendations for improvement.

### Latency and Workload

The COV Report for the period 1997-1999 expressed concerns about
latency, noting that: "general dissatisfaction was found, both in the
Committee and in the scientific constituency with the long delays now
experienced in processing proposals." In particular, the prior COV noted
that, due to increased workload, uneven distribution of submissions and
other factors the latency of the decision-making process was increasing
year-over-year as documented in Figure 5a below.

The current COV found improvement in latency, as shown in Figure 5b
below. This improvement is attributed to more effective use of the
Fast-Lane submission processes and increased attention to managing the
panel review process.

![](media/image6.wmf){width="3.7395833333333335in"
height="2.0416666666666665in"}

Figure 5a Latency for proposal processing FY 1997-1999

![](media/image7.wmf){width="3.7395833333333335in"
height="2.0416666666666665in"}

Figure 5b Latency for proposal processing FY2000-2002

One source of latency in the review process is program management
workload. Figure 6 below shows the year-on-year increase in CCR
competitive actions.

![](media/image8.wmf){width="5.965972222222222in"
height="3.3784722222222223in"}

Figure 6. Distribution of CCR Actions

This represents a 29% increase in actions over the review period. In
July 2000, CCR employed eight program officers to act on 1,190
proposals. As of June 2003, CCR used 11.5 program officers to make 1,540
competitive decisions in FY 2002. Thus, during the review period the
actual program officer workload has decreased to less than 134 actions
per program officer from the FY2000 level of 149, a 10% decrease in
workload. The Foundation is commended for providing this much-needed
relief to CCR program staff. The committee notes the salutary effect
this workload reduction has had on CCR productivity.

### Management of Merit Review Process

The Merit Review Process is being effectively managed and is of high
quality. A number of subcommittees, however, noted that in several
cases, reviewer selection resulted in panels that did not contain
experts in critical areas. Section 2 of this report details these
instances.

### Award Portfolio

The overall portfolio of awards appears to be appropriate for the
community. The committee notes, however, that CCR program officers
seldom use all of the award types available to them. For example, the
SGER award, which allows program officers considerable latitude in
funding high risk or unusual proposals and which seems to the committee
to be an important aspect of the award portfolio, is almost never used.
The committee is concerned that without using all of the tools at its
disposal, CCR will make only the most conservative awards that match
recommendations of panels.

The committee notes, however, that, despite the priority given to CISE
in recent years (see Figure 7 to compare budget increases for CISE
compared to the physical and life sciences), the level of investment in
the areas represented by CISE continues to lag other critical science
and technology investments. Figure 8 shows the year-on-year increase in
the CISE budget. Figure 9 compares the percentage of the Foundation
budget allocated to all activities over the review period.

![](media/image9.wmf){width="5.114583333333333in"
height="2.9791666666666665in"}

Figure 7. Comparative CISE Budget Increases

> ![](media/image10.wmf){width="4.771527777777778in"
> height="2.7111111111111112in"}

Figure 8. CISE Budget Increases

![](media/image11.wmf){width="5.114583333333333in"
height="2.9791666666666665in"}

Figure 9. Proportion of Research Budget allocated to Research Divisions

The Communications Research Subcommittee made special note of
disparities in funding networking and communications research in light
of the disappearance of many industrial research facilities and the
significant impact that telecommunications has on the nation's economy:

> The COV Communications Research (CR) subcommittee is highly concerned
> that the area of communications may not be properly represented within
> NSF. This concern has been existent in the communications engineering
> community for longer than a decade and the current organizational
> structure of CISE does not seem to be best suitable in serving this
> need. Communications technology is a highly critical one and the US
> leadership of the world in this area is extremely important to the
> nation. This leadership position is currently threatened by other
> countries and consortiums formed by countries. A prime example of a
> lost opportunity is in cellular voice network technology where a
> consortium of European companies, under the support of the European
> Union, developed a digital cellular voice communication standard known
> as GSM. This standard is being used in a large portion of the world
> today, including some major US markets. With the loss of the leading
> industrial research laboratories (Bell Labs, Bell Communications
> Research, AT&T Shannon Labs, IBM T. J. Watson Research Center, etc),
> the possibility of a similar leadership loss in new telecommunications
> technologies is a serious concern among technologists.

### Program Impact

Programs continue to be responsive to emerging trends and in some cases
to anticipate trends. The following are areas in which CCR anticipated
those emerging trends.

-   **Computing beyond silicon:** The famous law attributed to Intel
    co-founder Gordon Moore and which states that the cost of equivalent
    silicon-based computing power is halved (or equivalently the
    performance is doubled) every 18 months has driven the information
    technology revolution for forty years. Moore's Law will, however,
    "run out of steam" early in this century for both physical and
    economic reasons. CCR has been visionary in fostering theory and
    practice in new technologies beyond silicon and thus is an essential
    partner to industry and government in insuring the Nation's
    continued pre-eminence in high technology.

-   **Trustworthy Computing:** The recent emergence of trust and
    security as critical factors in our nation's cyber infrastructure
    has prompted widespread public comment about the need for increased
    attention to trustworthy computing. CCR was instrumental in
    developing cryptographic and other modern methods for securing
    computing systems. With the formation of a program office to focus
    attention on trust and security, CCR anticipates the growth in
    importance of this field.

-   **Nanotechnology:** Advances in physics and chemistry enable the
    manipulation of matter at the atomic and subatomic level. This
    enables the creation of new classes of materials and machines. CCR
    has anticipated the emergence of this field with investments in
    quantum, biologic and molecular technologies, networking and
    software technologies that are suited to nanoscale devices and
    application areas that will propel economic and scientific growth in
    nanotechnology.

-   **Homeland Security**: CCR has made historical investments that have
    resulted in technologies uniquely suited to the problems of homeland
    defense. Among these technologies are sensor systems suited for
    detection of chemical or biological attacks, graphics and interface
    technology for surveillance and protection, data analysis and
    visualization systems for extracting threats and vulnerabilities
    from massive background data, and communications/signal processing
    technology for enabling first-responder capabilities in times of
    national emergency.

### Panel Dynamics

A number of committee members commented on the almost exclusive reliance
on panels and the extent to which group dynamics of panel can adversely
affect panel results.

The CSA subcommittee notes:

> However, panels may tend to emphasize the opinion of one panelist who
> happens to be the expert in the domain of the proposal under review.
> One panelist may influence the others.

Members of the TOC Subcommittee reported on the effective use of
extended cyber-meetings that some conference/symposium program
committees have adopted. By extending the debate over a period of
several weeks and using modern internet technology as meeting
infrastructure, more thoughtful deliberations can be achieved and the
negative impact of overly aggressive (or passive) panel members can be
neutralized. The committee recommends that CCR conduct, on an
experimental basis, one of its panels in this format.

#### 

#### **Implementation of Merit Review Criteria** 

### Reviewing Criteria

Examination of proposal jackets and award decisions indicate that both
panels and investigators appear to be confused about newer reviewing
criteria like "broader impact". In many cases, these criteria are new,
so that investigators have not yet been briefed on their significance.

Program officers should provide concrete examples of such criteria and
include the examples in briefings, letters, articles and other venues to
help educate both panels and investigators.

### Panel Results

Panel summaries are frequently too telegraphic to effectively
communicate the rationale for a program decision or to adequately
summarize panel deliberations. Often investigators are simply told that
their proposals were of high quality but were not funded. In reviewing
program decisions, the Director of CCR should make a renewed effort to
insure that useful decision information is communicated to
investigators.

#### 

#### **Selection of Reviewers** 

### Panel Selection

The Committee believes that industry is underrepresented in panels.
Computer science is notable for the high percentage of the research
community that is associated with industrial R&D.

Total Panelists: 1661

Undetermined affiliation: 266

Known affiliation: 1395

Of those of known affiliation:

Industry/commerce: 129 (9.2%)

Government: 39 (2.8%)

Academic, nonprofit research: 1228 (99%)

> These numbers are fairly complete for some of the programs but not for
> others. Trusted Computing had over 30% industrial panelists, software
> engineering 15%, but the data is too sparse on some of the other
> programs to make a breakdown worthwhile.

CCR decision-making is enhanced if panels are knowledgeable about
industry developments and trends. By the same token, industrial
researchers who are informed about academic research trends and results
might be better able to anticipate fundamental shifts enabled by new
scientific developments.

The committee believes there are a number of factors to be addressed in
improving industrial participation in CCR panels. Industrial R&D
investment is broadly deployed between very near-term product
development and long-term basic investigation of the sort supported by
NSF. Recent economic trends in the IT industry have reduced that amount
of money that industry can spend on long-term research. For example,
central research laboratories such as Bell Labs and Digital Equipment's
research labs have either disappeared or sharply curtailed their
activities. As a result, relatively little is known about industrial
stakeholders in long-term research enterprise. Better understanding of
true industry stakeholders is needed to encourage more effective and
widespread participation.

The current practice of holding panels in the Washington DC area also
limits effective industrial participation because it naturally skews
panel membership toward the East Coast. The industrial base for the IT
industry is not centered on the East coast and limited travel budgets in
industrial research labs create hardships for industrial scientists who
must travel to Washington to participate.

The Committee recommends that CCR experiments with a limited number of
more "central" panel venues to determine the extent to which panel
membership is affected by meeting location. Universities and research
lab facilities can be made available to host such meetings, thus keeping
costs low.

The Committee notes that CCR has made a commendable attempt to align
gender participation in panels (currently 11% of reviewers, see Figure
10) with the number of female PI's who receive awards (currently 9%, see
Figure 11). The committee notes however, that as the number of female
panelists grows, there is a danger of increasing the workload of females
in the scientific community who actually serve on panels. The committee
recommends that NSF continue to actively seek out female reviewers from
outside the currently funded investigators (e.g., from industrial and
government laboratories) to avoid overloading female PI's with panel and
reviewing.

![](media/image12.wmf){width="5.0777777777777775in"
height="2.879861111111111in"}

Figure 10. Panelist Distribution by Gender

![](media/image13.wmf){width="4.780555555555556in"
height="2.7111111111111112in"}

Figure 11. Award Distributions by Gender

#### **Resulting Portfolio of Awards**

### Level of Funding and use of Available Resources

Figure 12 shows the average CCR proposal request versus actual awards.
It is obvious from the data that, although investigators' requests have,
over time, increased to account for inflationary effects, increased
sophistication of laboratories and experiments, number of researchers
and other factors, CCR program officers have held award size more or
less constant. A possible interpretation of this trend is that program
officers are following a "rote" standard for CCR awards, in an attempt
to spread an inadequate budget over as many projects as possible. The
Committee does not believe that the community is well-served by such a
strategy.

[The Committee recommends that CCR management undertake a review of
funding levels to determine whether or not the needs of funded proposals
are actually being met by such practices.]{.underline}

![](media/image14.wmf){width="5.594444444444444in"
height="3.1666666666666665in"}

Figure 12. Grant Size Versus Request

Information about an investigator's overall funding would help program
officers make informed decisions about the effect of funding levels on
the proposed research. For example, while \$100K standalone support for
a senior investigator may not be adequate to carry out an effective
experimental program, a 20% marginal increase to an existing funding
base may be important incremental funding to a laboratory. In these
instances, a \$100K award in the first case will not be useful but in
the latter case will contribute to the overall success of the research.
This funding information is not currently provided to program officers,
although the information is collected from investigators in proposals.
The Committee recommends that CISE collect this information, provide it
for use by Program Officers and that CCR management use this information
in determining appropriateness of funding levels for selected proposals.

#### **Management of Programs**

### Program Planning and Prioritization

Program planning and prioritization appear to be uneven in CCR. Program
officers who briefed the Committee did not always exhibit knowledge of
priorities within CCR, CISE and NSF. Greater transparency in program
planning and spending priorities would be valuable.

Individual subcommittees reported varying degrees of satisfaction in the
effectiveness of fostering collaboration between clusters and program.
Trust and security, for example, are cross-cutting issues that all
programs should be aware of. On the other hand communications,
networking and digital signal processing have historically benefited
from collaborative research. Special efforts within CCR and CISE to
identify and focus priorities on such cross-cutting technologies would
be valuable.

The SEL Subcommittee, in particular, notes:

> Given the nature of the NSF structure, many big-payoff research areas
> require the interaction of several PDs. This creates a problem for a
> principal investigator -- where to go with a new proposal, a problem
> for creating panels -- how do you create the proper mix and motivate
> them to see the other sides of the issues. It is hard to motivate a
> short term interdisciplinary activity without imposing a fair amount
> of interaction and work on the appropriate PDs. But it appears
> difficult to add another time-consuming activity to the already
> over-burdened PDs. (This interaction is especially relevant for an
> enabling discipline like software engineering given the pervasive
> nature of software to every scientific discipline.)

### Financial Management

As shown in Figure 13, the number of "continuing awards" is rising
dramatically. This decreases the ability of program officers to make
effective funding decisions in his/her first year because of previous
obligations. Program officers on 2-year IPA appointments complain of
their inability to make an impact on program decisions because of
commitments made by prior program officers. Continuing awards are a
"mortgage" on the future that harms the long-term health of CCR if they
are the dominant award type. The Committee recommends that CCR increase
the percentage of standard grants.

![](media/image15.wmf){width="4.9625in" height="2.810416666666667in"}

Figure 13. Award Type

The Committee heard on a number of occasions that some program officers
are confused about budget data that is traditionally supplied to CCR. It
is the committee's conclusion that either the data is not supplied in a
timely way by NSF administration or the Program Managers are not
adequately informed. In either event, intervention on the part of CISE
management seems to be needed.

### Program Continuity

The Committee noted the positive effects that CAREER awards and ITR's
have had on the computer science community. CAREER awards have enabled
the most promising new researchers to begin their careers with adequate
support which in turn has been essential in energizing a new generation
of researchers. A natural question is what will happen to these
investigators after the CAREER awards expire. The committee urges CCR to
develop a strategy to deal with follow-on plans for these awards.

ITR's have had a dramatic impact on computer science research and have
fostered effective collaboration in ways that would have been difficult
to achieve otherwise. As shown in Figure 14, virtually all of the
increase in CCR's awards is due to the ITR awards. On the other hand,
the award rate for the ITR program has dramatically outstripped the
declining award rate for the regular program (see Figure 15).

![](media/image16.wmf){width="5.6in" height="3.171527777777778in"}

Figure 14. Award Type

![](media/image3.wmf){width="5.927083333333333in"
height="3.3666666666666667in"}

Figure 15. Award Rates

The effect on the computer science community has been more pronounced
than for other program areas at NSF. Several Committee members
recounted, for example, the extent to which their own departments have
succeeded primarily because of the ITR program. The question that is
unresolved is how CCR will continue to derive value from its ITR
investments after the program expires next year. The Committee
encourages CCR and CISE to develop a strategy now for minimizing the
adverse effects of ITR expiration on the field.

### Output and Outcomes of NSF Investments

CCR research program investments are successful if they result in the
following outcomes:

-   People: development of a diverse, competitive, globally engaged
    workforce of scientists, engineers and well-prepared citizens

-   Ideas: discovery across the frontiers of science and engineering
    connected to learning, innovations and service to society

-   Tools: development of broadly accessible, state-of-the are research
    and education tools.

One area in which the Committee sees need for improvement is the CCR
focus on doing more to encourage diversity in the research community.
The Committee is not satisfied with the award rate for females and
ethnic minorities in CCR programs and recommends a renewed emphasis on
programs and processes that result in a more diverse community.

The Committee identified many examples of successful outcomes of NSF
investment in CCR programs and investigators.

#### **Biocomputation**

> Insight into the behavior of large organic molecules such as DNA can
> be had with the help of computers, which provide the only feasible
> approach to exploring the combinatorially large spaces of possible
> configurations for those molecules. Martin Farach-Colton, Rutgers,
> CCR-98020879, has applied the theory of interval graphs to the
> development of a fast algorithm for assembling DNA sequences. He has
> used this algorithm to detect errors in the data from the Human Genome
> Project. Thanks to the superior speed of this algorithm relative to
> those developed at UC Santa Cruz and NIH, Farach-Colton has been able
> to produce the most reliable assembly of the human genome to date.

#### **Internet Behavior**

> The explosive impact of the Internet on both commerce and our personal
> lives has called for a deeper understanding of its behavior. Studies
> to date of the Internet have modeled its statistical aspects by
> assuming an independence of events that would suggest classical
> exponential distributions of the kind associated with Poisson
> processes. In 1999 Falutsous et al questioned this assumption and
> found that in fact events in the Internet followed distributions of a
> subexponential kind that had previously been associated with such
> areas as natural language, communication channels, and chaos theory.
> Christos Papadimitriou, UC Berkeley, CCR-9820897, asked why this
> should be, and came up with a strikingly simple explanation in terms
> of conflicting constraints on events which must be reconciled in some
> near-optimal way.

#### **The Simplex Method**

> During the last two years Shang-Hua Teng[^1] (U of Illinois at Urbana
> Champagne), together with Dan Spielman (MIT), solved a
> long-outstanding open question in Mathematical Programming,
> Optimization, and Theoretical Computer Science. They proved that the
> Simplex Method for Linear Programming usually takes polynomial number
> of steps. In the process, they developed a new algorithm-analysis
> framework that we call smoothed analysis that can help explain the
> success of many algorithms and heuristics that the traditional
> algorithm-analysis frameworks, such as the worse-case and average-case
> analysis, cannot. The simplex algorithm is the classic example of an
> algorithm that is known to perform well in practice but which takes
> exponential time in the worst case. It has been an active subject for
> both mathematical and experimental studies for more than 50 years.

#### **High Performance Computing Techniques for Scientific Problems**

> The NSF awards to Jun Zhang[^2], U of Kentucky, support research and
> development of high performance computational techniques for solving
> sparse linear systems on parallel and distributed computers with
> scientific, engineering, and industrial applications. Under the
> support of these NSF grants, the Laboratory for High Performance
> Scientific Computing and Computer Simulation (the HiPSCCS Lab) at the
> University of Kentucky has developed new robust iterative methods for
> solving difficult sparse matrices on parallel and distributed
> computers.
>
> Zhang's new techniques have caught the attention of some application
> scientists. Research and development engineers at the General Motors
> Company expressed interest in using the parallel linear system solvers
> developed at the HiPSCCS Lab in their car engine combustion code to
> speed-up their large scale computer simulation in designing new cars.
> Researchers at the Japanese Research Organization for Information
> Science & Technology (RIST) is interested in utilizing our high
> performance linear system solvers in conducting very large scale
> computer simulation of solid Earth. Their goal is to attempt more
> accurate predication of earthquakes. Thus, results from the NSF funded
> research projects have the potential to make people\'s life better by
> designing new and safer cars, and to save human lives by predicting
> natural disasters more accurately.

#### **Development of the Research Community**

> The PI's supported by the CSA program have received numerous awards,
> such the Alfred P. Sloan Research Fellowships (To Profs. Sarita Adve,
> Doug Burger and Steve Keckler). A PI has become an IEEE fellow (Prof.
> Sachin Sapatekhar),. PIs have also received best paper awards in
> international conferences and have acquired many patents as a result
> of their research. Also, the subcommittee found that CSA program has
> increased the REU (Research Experience for Undergraduate students)
> awards to 8 in 2002, which we feel is a positive encouragement for the
> participation of undergraduate students in research activities.

#### **Tools to Accelerate Research in new Hardware Architectures**

> The Liberty Research Group at Princeton University has developed the
> Liberty Simulation Environment (LSE). LSE is a simulator builder
> designed to reduce barriers in the research of radical hardware
> systems by reducing the cost and improving the fidelity of simulation
> models. LSE is already in use in academic and industrial labs
> including UC Berkeley, Rice, UIUC, Princeton, Intel, and Infineon.
> More information can be found at <http://liberty.princeton.edu/>

#### **Tools to Enhance Personal Privacy in Information Systems**

> Annie Antón; North Carolina State University; CCR-9983926; has
> developed a set of tools that help ensure that privacy policies are
> aligned with the software systems that they govern. One such tool to
> support the discovery, elaboration and management of system use
> scenarios for the validation of software requirements and policies, is
> SMaRT (Scenario Management and Requirements Tool).

#### **Giga-Scale Systems on a Chip**

> Supported under the NSF Award CCR-0096383, the researchers at UCLA led
> by Prof. Jason Cong have made significant progress on optimality study
> of existing placement algorithms for integrated circuits. Using a set
> of cleverly constructed circuit examples that match industrial circuit
> characteristics and have known optimal solutions, researchers from
> UCLA show that the results of leading placement tools from both
> industry and academia (including, for example, Qplace from Cadence)
> are 70% to 150% (!) away from the optimal solutions in terms of the
> total wirelength. This is the first time a quantitative evaluation is
> provided for the optimality of existing placement algorithms. Note
> that if such a large optimality gap can be closed, it will be
> equivalent to several technology generation advancements (in
> comparison, the introduction of copper interconnects is equivalent to
> a 30% wirelength reduction, and so is each process technology scaling,
> but each requires *multi-billion dollar* investment). This research
> signifies the opportunity and importance of further research in
> physical design for large-scale integrated circuits.
>
> This result was first reported at ASPDAC\'2003 in Japan.  It soon
> generated great interest from both the research community and
> semiconductor/EDA industry. It was featured as the cover story in the
> EE Times, a leading trade journal, on February 5, 2003. The examples
> used in this study (named PEKO examples) have been made available on
> the Internet at
> [http://cadlab.cs.ucla.edu/\~pubbench/placement/](http://ballade.cs.ucla.edu/~pubbench/placement/)
> and have been downloaded by researchers and engineers from over 40
> companies and universities worldwide within one week since the result
> was covered by EE Times.

#### **Publicly Distributed Collaboration Technologies**

> CCR 9980616, Toby Berger. Open source desktop/laptop video
> conferencing software called qvix/cu30 for free noncommercial use.
> Available from <http://www.sourceforge.net/projects/cu30>.

#### **Biomedical Imaging**

> Bouman and Low, Fluorescence Optical Diffusion Tomography work from
> Purdue -- an example of multidisciplinary work with a colleague in
> Chemistry. Bouman has partnered with Phil Low who is examining methods
> for detecting tumors using fluorescein conjugate contrast agents.
> Bouman et al. have developed a fast and simple method that can image
> fluorescing tumors obscured by more than a centimeter of tissue.

#### **Engineering and Technology in K-12 Education**

> The Infinity Project --- "The Infinity Project is a nationally
> recognized partnership between leading research universities,
> industry, government, and K-12 educators to help school districts
> incorporate modern engineering and technology in their high school
> curricula." Texas Instruments has provided \$1 million in support of
> the project. Work on the Project has been underway during the full
> three year interval over which this review applies. It is now being
> taught in more than 70 high schools nationwide with significant growth
> planned for the future. It teaches engineering and technical concepts
> using basic examples from the field of signal processing systems
> (SPS).
>
> The Project has been organized by the signal processing community and,
> in particular, many of the leaders of the project are current and past
> SPS awardees. Specifically, the director of the project is Geoffrey
> Orsak of SMU, and the advisory panel includes Don Johnson, Jerry
> Gibson, Delores Etter, and Leah Jamieson. The co-authors of the
> educational material include Dave Munson, a current awardee, and Scott
> Douglas, a former SPS CAREER awardee. Sally Wood, another co-author,
> served for many years on the NSF's Committee for Woman, Minorities,
> and the Handicapped.

#### **Outreach to Under-represented Minority Students**

> The Summer Internship Program in Hybrid and Embedded Software
> (SIPHER), which is currently run by Vanderbilt University and
> University of California Berkeley as part of their cooperative
> research agreement (CCR-0225610) "Foundations of Hybrid and Embedded
> Software Systems." The SIPHER program is organized around the outreach
> effort toward HBCUs with a critical mass of minority students and
> community colleges both in the South and in California. Dr. Brian
> Williams, Assistant Dean for Student Affairs at Vanderbilt University,
> is establishing linkages to schools like Kisk University, Tennessee
> State University, University of Alabama-Birmingham, Morehouse College,
> Spelman College, Florida State University, Florida A&M University and
> others. The SIPHER project is profoundly changing the current
> curriculums in embedded and hybrid systems at Vanderbilt University
> and UC Berkeley.

##  Comments on the COV Process

In plenary session, the Committee discussed the COV Process itself. CCR
program and support staffs were well-prepared for this review and
provided both the Chair and the Committee as a whole a number of helpful
tools. These include report templates and extensive documentation.
However, the Committee did not several areas in which improved processes
and support capabilities would have significantly improved COV ability
to review programs.

### More lead time for COV Meeting

The lead time allowed for a COV of this caliber was judged by most
participants to be inadequate. COV reviews are regular events at NSF and
the CCR management is well aware of when a COV will be scheduled.
Building this into an annual calendar that permits selection of
committee members and chair six to nine months in advance is not
unreasonable.

### More effective briefings by NSF staff on programs

As noted above, Program officers were uneven in their preparation for
this review. Formal briefings outlining program status and strategy
would have been helpful. For the most part, program officers were
content to field questions from panelists but did adequate provide a
context in which COV members could work.

### Better guidance for COV on format and content of report

The COV Chair was the only member of the Committee who received an
example of a prior COV report. As a result, most committee members were
in the dark about what kind of report they were being asked to produce.

### Better collaborative tools at the COV meetings to facilitate production of the report

The meeting rooms and facilities were inadequately equipped for the kind
of collaborative report production needed for a COV review. Program
staff were not adequately trained on the use of A/V facilities and
overhead projection materials were either scarce or of low quality.

######### 

# The COV Review Process

##  COV Charter and Membership

In compliance with the three-year review requirement, the National
Science Foundation's Division for Computer-Communications Research (CCR)
convened a Committee of Visitors (COV) for two days, June 5 and 6, 2003,
at NSF headquarters, Arlington, VA. The "COV" is a singularly important
committee convened once every three years to assess the effectiveness of
the program under review. The Committee is composed of selected senior
researchers in relevant specialty fields and was chosen to span the
scientific purview of the CCR Division. At the outset, the Committee was
provided comprehensive documentation for its two-day meeting, including
succinct description of the charge to the Committee and its obligations
in responding.

The Committee of Visitors to CCR in June 2003 was made up of the chair,
Richard DeMillo of Georgia Institute of Technology, and 2 to 4 members
for each of the 10 "core" programs in CCR. The composition of the
committee according to the programs under review is as follows. (Names,
affiliations, and contact information for the members is found in
Appendix 2

1.  Theory of Computing (TOC)\
    Juris Hartmanis, Vaughan Pratt, Robert Sedgewick

2.  Graphics, Symbolic and Geometric Computation (GSG)\
    George Labahn, Ahmed Sameh, Jessica K. Hodgins, Chandrajit Bajaj

3.  Computer Systems Architecture (CSA)\
    Jean-Luc Gaudiot, Pen-Chung Yew, Kemal Ebcioglu

4.  Operating Systems and Compilers (OSC)\
    Marion Harmon, Kishore Ramachandran, Hank Levy

5.  Software Engineering and Languages (SEL)\
    Victor R. Basili, Debra J. Richardson, Tony Wasserman

6.  Design Automation for Micro and Nano Systems (DA)\
    Joseph R. Cavallaro, Robert W. Newcomb, Yannis P. Tsividis

7.  Communications Research (COMM)\
    Arlene A. Cole-Rhodes, Ender Ayanoglu, Chris Heegard

8.  Signal Processing Systems (SPS)\
    Mark Smith, John Treichler, Ron Schafer

9.  Embedded and Hybrid Systems (EHS)\
    Kane Kim, Douglas C. Schmidt

10. Trusted Computing (TC)\
    George Cybenko, Peter G. Neumann

Expertise on the COV was selected to match the topic areas, and the ten
sub-committees were charged with studying their respective program
areas. After breaking into the ten separate sub-committees, these groups
were provided extensive proposal documentation, as well as individual
conference rooms in which to conduct their studies

##  Charge to the COV

The COV report should provide a balanced assessment of NSF's performance
in two primary areas: (A) the integrity and efficiency of the
***processes*** related to proposal review; and (B) the quality of the
***results*** of NSF's investments in the form of outputs and outcomes
that appear over time. The COV also explores the relationships between
award decisions and program/NSF-wide goals in order to determine the
likelihood that the portfolio will lead to the desired results in the
future. Discussions leading to answers for Part A of the Core Questions
will require study of confidential material such as declined proposals
and reviewer comments. *COV reports should not contain confidential
material or specific information about declined proposals**.***
Discussions leading to answers for Part B of the Core Questions will
involve study of non-confidential material such as results of NSF-funded
projects. It is important to recognize that the reports generated by
COV\'s are used in assessing agency progress in order to meet
government-wide performance reporting requirements, and are made
available to the public. Since material from COV reports is used in NSF
performance reports, the COV report may be subject to an audit.

*We encourage COV members to provide comments to NSF on how to improve
in all areas, as well as suggestions for the COV process, format, and
questions.*

##  Methodology 

Each sub-committee was instructed to respond to the template items with
the intent of revealing those points where consensus can be achieved
across the whole committee. The agenda of the two-day meeting was
designed so that periodic plenary sessions could coalesce topics from
the sub-committees for group analyses, for comparisons of status of the
respective studies, and for identification of common issues. The FY2003
Template is reproduced in Appendix 4.

  --------- -------------------------------------------------------------
  Section   

  2         
  --------- -------------------------------------------------------------

Subcommittee Reports

# Theory of Computing (TOC)

##### **FY 2003 REPORT TEMPLATE FOR**

#####  **NSF COMMITTEES OF VISITORS (COVs)**

  -----------------------------------------------------------------------
  **Date of COV: June 5-6, 2003**

  **Program/Cluster: TOC**

  **Division: CCR**

  **Directorate: CISE**

  **Number of actions reviewed by COV**[^3]**:\
  Awards: 75 Declinations: 119 Other: 4**

  **Total number of actions within Program/Cluster/Division during period
  being reviewed by COV**[^4]**: Awards: Declinations: Other:**

  **Manner in which reviewed actions were selected:\
  **
  -----------------------------------------------------------------------

**A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

**A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures..**

+-----------------------------------------------------------+----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES**  | **YES,   |
|                                                           | NO,**    |
|                                                           |          |
|                                                           | **DATA   |
|                                                           | NOT      |
|                                                           | AV       |
|                                                           | AILABLE, |
|                                                           | or NOT   |
|                                                           | APPL     |
|                                                           | ICABLE** |
+-----------------------------------------------------------+----------+
| Is the review mechanism appropriate? (panels, ad hoc      | Yes      |
| reviews, site visits)                                     |          |
|                                                           |          |
| Comments: The primary review mechanism for TOC is the     |          |
| panel, with an occasional independent review.             |          |
+-----------------------------------------------------------+----------+
| Is the review process efficient and effective?            | Yes      |
|                                                           |          |
| Comments: The panels make the reviews both timely and     |          |
| efficient. The subcommittee felt that the move to panels  |          |
| mirrored the traditional organization of program          |          |
| committees, which used to meet for a two-day marathon of  |          |
| sorting out the accepts from the rejects. The recent      |          |
| trend to two-week electronically mediated PC committees   |          |
| might be a good way for NSF to organize its panels in the |          |
| future. One benefit is to dilute the impact of            |          |
| intimidation as a negotiating tool. Another is to permit  |          |
| a much longer window of deliberation, avoiding the        |          |
| mistakes that often result from a premature rush to       |          |
| judgment.                                                 |          |
+-----------------------------------------------------------+----------+
| Are reviews consistent with priorities and criteria       | Yes      |
| stated in the program's solicitations, announcements, and |          |
| guidelines?                                               |          |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Do the individual reviews (either mail or panel) provide  | No       |
| sufficient information for the principal investigator(s)  |          |
| to understand the basis for the reviewer's                |          |
| recommendation?                                           |          |
|                                                           |          |
| Comments: The individual reviews tend to be cryptically   |          |
| brief, giving the PI little idea as to what to do         |          |
| differently next time. TOC is a notable and repeat        |          |
| offender in this respect: theorists seem to assume that   |          |
| their colleagues are mind readers.                        |          |
+-----------------------------------------------------------+----------+
| Do the panel summaries provide sufficient information for | No       |
| the principal investigator(s) to understand the basis for |          |
| the panel recommendation?\                                |          |
| Comments: The panel summaries add too little to the       |          |
| individual reviews to compensate for their brevity.       |          |
+-----------------------------------------------------------+----------+
| Is the documentation for recommendations complete, and    | Yes      |
| does the program officer provide sufficient information   |          |
| and justification for her/his recommendation?             |          |
|                                                           |          |
| Comments: Recommendation documentation is helped a lot by |          |
| having a form to be filled in.                            |          |
+-----------------------------------------------------------+----------+
| Is the time to decision appropriate?                      | Yes      |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Discuss issues identified by the COV concerning the       |          |
| quality and effectiveness of the program's use of merit   |          |
| review procedures:                                        |          |
|                                                           |          |
| More explicit feedback would be helpful in the individual |          |
| reviews and panel summaries. The effectiveness of the F7  |          |
| forms in this regard suggests that more detailed          |          |
| templates for the reviews and panels might encourage more |          |
| thoughtful feedback.                                      |          |
+-----------------------------------------------------------+----------+

**A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers.**

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       | No        |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments:                                                |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     | No        |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments:                                                |           |
+----------------------------------------------------------+-----------+
| Have the review analyses (Form 7s) addressed whether the | No        |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments:                                                |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system.               |           |
|                                                          |           |
| The subcommittee is satisfied that Criterion 1, "What is |           |
| the intellectual merit of the proposed activity?" of the |           |
| new NSF Merit Review Criteria is being fully addressed   |           |
| in the review of TOC proposals. Although Criterion 2,    |           |
| "What are the broader impacts of the proposed            |           |
| activity?", is honored more in the breach, we are        |           |
| convinced that the TOC program has and is funding        |           |
| research with a broad and powerful positive educational  |           |
| and societal effects, as documented below. Where the     |           |
| implications are considered at all, they tend to be      |           |
| viewed very narrowly.                                    |           |
+----------------------------------------------------------+-----------+

**A.3 Questions concerning the selection of reviewers.**

+---------------------------------------------------------+------------+
| **Selection of Reviewers**                              | **YES,     |
|                                                         | NO,**      |
|                                                         |            |
|                                                         | **DATA NOT |
|                                                         | AVAILABLE, |
|                                                         | or NOT     |
|                                                         | AP         |
|                                                         | PLICABLE** |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       | Yes        |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            | Yes        |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    | Yes        |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      | Yes        |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
|                                                         |            |
| The TOC program is to be commended for its insightfully |            |
| fair selection of reviewers. The subcommittee is        |            |
| satisfied that the reviewers are broadly representative |            |
| of the field, large and small institutions, women and   |            |
| minorities. We are also satisfied that conflicts of     |            |
| interest are taken seriously and are properly handled.  |            |
+---------------------------------------------------------+------------+

**A.4 Questions concerning the resulting portfolio of awards under
review.**

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     | Appropriate   |
| projects supported by the program.                   |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  | Appropriate   |
| scope of the projects?                               |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The size of the awards is appropriate given the      |               |
| limits of available funding.                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| High Risk Proposals?                                 |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| Multidisciplinary Proposals?                         |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| Innovative Proposals?                                |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| Funding for centers, groups and awards to            |               |
| individuals?                                         |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| Awards to new investigators?                         |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| Geographical distribution of Principal               |               |
| Investigators?                                       |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| Institutional types?                                 |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| Projects that integrate research and education?      |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance:                                             |               |
|                                                      |               |
| Across disciplines and subdisciplines of the         |               |
| activity and of emerging opportunities?              |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          | Appropriate   |
| participation of underrepresented groups?            |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      | Yes           |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments: Recent areas that are either of national   |               |
| importance or simply "hot" TOC topics include        |               |
| cryptographic algorithms (Bellare), Internet         |               |
| technology, biocomputing, and quantum cryptography.  |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
|                                                      |               |
| Theory of Computation maintains a healthy balance    |               |
| across all the factors asked about. The 2002 award   |               |
| to Immerman illustrates an award for a high risk     |               |
| proposal from a researcher with a very strong track  |               |
| record.                                              |               |
|                                                      |               |
| The program continues a trend of many years to       |               |
| support work of extremely high quality. A            |               |
| concomitant trend is the inadequacy of available     |               |
| funding to support all of the high quality           |               |
| proposals. A particularly disturbing trend is the    |               |
| increasing gap between the appropriateness and       |               |
| actual size of awards: while the allocation is fair  |               |
| given the available levels, the amounts are so       |               |
| inadequate as to verge on irrelevance.               |               |
+------------------------------------------------------+---------------+

**A.5 Management of the program under review**

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments: Overall the program is well managed.                        |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments: The program responds well to emerging research trends. Its  |
| response to emerging education trends is less clear, due largely to   |
| lack of appropriate proposal material to work with.                   |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments: Not a lot, but not much needed for TOC.                     |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| The program as a whole seems well managed. As far as responsiveness   |
| to emerging research trends is concerned, the theory community tends  |
| to take care of \`\`pushing the envelope'' and the subcommittee found |
| no indications of undue neglect of proposals tracking new research    |
| trends.                                                               |
|                                                                       |
| We did not find any similarly innovative proposals on the education   |
| front, which meant that the program is not responding to emerging     |
| education trends, but only because the theory community is not        |
| providing NSF with the necessary raw material in the form of such     |
| proposals.                                                            |
|                                                                       |
| Due to the relatively simple demands of research having paper and pen |
| as its primary tools, correspondingly few demands are placed on       |
| program planning and prioritization: not much is provided but not     |
| much is needed.                                                       |
|                                                                       |
| By far the greatest single concern relevant to the program management |
| is the woefully small amount of money available for allocation to the |
| proposed research. This financial shortage has two impacts. First     |
| there are good proposals that cannot be funded. Second, the amount of |
| a typical NSF TOC award is close to the level where the net overhead  |
| of administering each grant by all parties concerned exceeds the      |
| intended benefit of the money.                                        |
+-----------------------------------------------------------------------+

##### B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.**

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

+-----------------------------------------------------------------------+
| **[B.1 NSF OUTCOME GOAL for PEOPLE:]{.underline}** **Developing "a    |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| Comments: NSF is without peer as a US agency devoted to developing a  |
| competitive workforce of theoretical computer scientists. NSF's       |
| CAREER awards are of especial benefit here in assisting young         |
| theoreticians to focus on their research without the distraction of   |
| repeated "trips to the well." The theory community is itself highly   |
| competitive; NSF maintains the standards so created by reliably       |
| identifying the best of the bunch.                                    |
+-----------------------------------------------------------------------+
| **[B.2 NSF OUTCOME GOAL for IDEAS:]{.underline} Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| **Comments:**                                                         |
|                                                                       |
| 1.  **Biocomputation.\                                                |
|     **Insight into the behavior of large organic molecules such as    |
|     DNA can be had with the help of computers, which provide the only |
|     feasible approach to exploring the combinatorially large spaces   |
|     of possible configurations for those molecules. Martin            |
|     Farach-Colton, Rutgers, CCR-98020879, has applied the theory of   |
|     interval graphs to the development of a fast algorithm for        |
|     assembling DNA sequences. He has used this algorithm to detect    |
|     errors in the data from the Human Genome Project. Thanks to the   |
|     superior speed of this algorithm relative to those developed at   |
|     UC Santa Cruz and NIH, Farach-Colton has been able to produce the |
|     most reliable assembly of the human genome to date.**\            |
|     **                                                                |
|                                                                       |
| 2.  **Cryptographic Algorithms.\                                      |
|     **                                                                |
|                                                                       |
|     a.  Deterministic Cryptography.\                                  |
|         Cryptography, which in the US has traditionally been the      |
|         exclusive domain of the National Security Administration, has |
|         within the past quarter-century opened up to permit           |
|         contributions from academia, with the Rivest-Shamir-Adleman   |
|         (RSA) algorithm setting the pace in 1976. Daniele Micciancio, |
|         UC San Diego, CCR-0093029, has applied geometry and lattice   |
|         theory (of the kind used in number theory rather than that    |
|         used in algebraic logic) to the construction of secure        |
|         computing systems, in the process developing the strongest    |
|         known average case hardness result for lattice-based          |
|         problems.                                                     |
|                                                                       |
|     b.  Quantum Cryptography.\                                        |
|         Work by Charles Bennett and Gilles Brassard in the early      |
|         1980s spawned a variant of quantum computing applicable to    |
|         cryptography. Whereas quantum computing offers an exponential |
|         speedup over the known deterministic algorithms for at least  |
|         the problem of integer factoring (and so by implication       |
|         possibly for other problems), quantum cryptography offers     |
|         instead a communication channel that is guaranteed to detect  |
|         the presence of any eavesdropper. But unlike quantum          |
|         computing, whose promise has yet to be realized in practice   |
|         at useful scales, quantum cryptography is working today over  |
|         channels many kilometers long, making it eminently practical  |
|         now. Andy Yao (CCR-9820885) has developed an algorithm that   |
|         solves the consensus problem, if not exactly at least for all |
|         practical purposes, using quantum cryptography. Consensus is  |
|         demonstrably not achievable using eterministic computation.   |
|                                                                       |
| 3.  Internet Behavior.\                                               |
|     The explosive impact of the Internet on both commerce and our     |
|     personal lives has called for a deeper understanding of its       |
|     behavior. Studies to date of the Internet have modeled its        |
|     statistical aspects by assuming an independence of events that    |
|     would suggest classical exponential distributions of the kind     |
|     associated with Poisson processes. In 1999 Falutsous et al        |
|     questioned this assumption and found that in fact events in the   |
|     Internet followed distributions of a subexponential kind that had |
|     previously been associated with such areas as natural language,   |
|     communication channels, and chaos theory. Christos Papadimitriou, |
|     UC Berkeley, CCR-9820897, asked why this should be, and came up   |
|     with a strikingly simple explanation in terms of conflicting      |
|     constraints on events which must be reconciled in some            |
|     near-optimal way.                                                 |
+-----------------------------------------------------------------------+
| **B.3 OUTCOME GOAL for TOOLS: Providing "broadly accessible,          |
| state-of-the-art and shared research and education tools."**          |
|                                                                       |
| Comments: Not applicable (Theory of Computation is much less          |
| tool-oriented than other CCR programs).                               |
+-----------------------------------------------------------------------+

#####  **C. OTHER TOPICS**

**C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.**

No gaps.

**C.2 Please provide comments as appropriate on the program's
performance in meeting program-specific goals and objectives that are
not covered by the above questions.**

The program is performing well given its available resources.
Unfortunately those resources are so small as to undermine the influence
of the program.

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

None

**C.4 Please provide comments on any other issues the COV feels are
relevant.**

None.

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

Projects funded prior to the evaluation period may generate nuggets
during the evaluation period. A list of those earlier projects would be
helpful in identifying such evaluation-period nuggets attributable to
earlier NSF funding. Otherwise these "late nuggets" will fall through
this crack in the COV review process.

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the Theory of Computing (TOC)

\[Juris Hartmanis\]

Chair

# Graphics, Symbolic and Geometric Computation (GSC)

##### **FY 2003 REPORT TEMPLATE FOR**

#####  **NSF COMMITTEES OF VISITORS (COVs)**

  -----------------------------------------------------------------------
  **Date of COV: June 5-6, 2003**

  **Program/Cluster: Graphics, Symbolic and Geometric Computation**

  **Division: CCR**

  **Directorate: CISE**

  **Number of actions reviewed by COV**[^5]**:\
  Awards: Declinations: Other:**

  **Total number of actions within Program/Cluster/Division during period
  being reviewed by COV**[^6]**: Awards: Declinations: Other:**

  **Manner in which reviewed actions were selected:\
  **
  -----------------------------------------------------------------------

**A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

1.  **Effectiveness of the program's use of merit review procedures:**

> a\. Overall design, including appropriateness of review mechanism
> (panels, ad hoc reviews, site visits);
>
> b\. Effectiveness of program's review process;

c.  Efficiency; time to decision;

d.  Completeness of documentation making recommendations;

e.  Consistency with priorities and criteria stated in the program's
    solicitations, announcements, and guidelines.

**A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures..**

+----------------------------------------------------------+-----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES** | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Is the review mechanism appropriate? (panels, ad hoc     |           |
| reviews, site visits)                                    |           |
|                                                          |           |
| Comments: We believe that the panel system is effective  |           |
| for many reasons but particularly because it allows a    |           |
| program director to judge both the proposal and the      |           |
| reviewers doing a review in one fell swoop. It typically |           |
| shows quickly which reviewer has expertise to comment of |           |
| the technical points of a given proposal. It also        |           |
| effectively shows which reviews have some depth versus   |           |
| which reviews are just quick readings of a proposal done |           |
| without enough care or thought. Finally it also ensures  |           |
| that reviews are done in a timely manner                 |           |
+----------------------------------------------------------+-----------+
| Is the review process efficient and effective?           |           |
|                                                          |           |
| Comments: The review process is efficient and effective, |           |
| although it would be better if the time from the         |           |
| submission deadline to the panel meeting could be three  |           |
| months rather than four.                                 |           |
+----------------------------------------------------------+-----------+
| Are reviews consistent with priorities and criteria      |           |
| stated in the program's solicitations, announcements,    |           |
| and guidelines?                                          |           |
|                                                          |           |
| Comments: The time to decision is usually appropriate.   |           |
| The numbers that we saw were 4-10 months (average of 10  |           |
| months for numerical computing). It would be better if   |           |
| the 10 months could be reduced to 8 but given the        |           |
| vagaries of the NSF budget and the resulting delays,     |           |
| that improvement seems unlikely.                         |           |
+----------------------------------------------------------+-----------+
| Do the individual reviews (either mail or panel) provide |           |
| sufficient information for the principal investigator(s) |           |
| to understand the basis for the reviewer's               |           |
| recommendation?                                          |           |
|                                                          |           |
| Comments: The committee noted that panels are the        |           |
| primary form of review. We saw very few ad hoc (mail)    |           |
| reviews in the folders that we reviewed.                 |           |
+----------------------------------------------------------+-----------+
| Do the panel summaries provide sufficient information    |           |
| for the principal investigator(s) to understand the      |           |
| basis for the panel recommendation?\                     |           |
| Comments: While the panels are expedient and allow for   |           |
| calibration they sometimes do have some disadvantage. In |           |
| particular, the reviews are often written by people less |           |
| expert in the topic of the proposal than mail reviewers  |           |
| picked for an individual proposal would have been. As an |           |
| example, we saw one funded animation proposal that was   |           |
| reviewed by no one with particular expertise in          |           |
| animation and no animation experts were on the panel. We |           |
| suggest that a greater use of ad hoc reviews might       |           |
| improve the process. If those needs could be anticipated |           |
| in advance then the mail reviews could occur             |           |
| concurrently with the panel reviews and be taken into    |           |
| consideration at the panel meeting. In addition, the     |           |
| program manager needs to be particularly careful that    |           |
| the experts who are on the panel for a particular        |           |
| proposal are not also conflicted with that proposal and  |           |
| therefore unable to participate.                         |           |
+----------------------------------------------------------+-----------+
| Is the documentation for recommendations complete, and   |           |
| does the program officer provide sufficient information  |           |
| and justification for her/his recommendation?            |           |
|                                                          |           |
| Comments: Electronic reviews available to panelists      |           |
| before meetings are highly useful and should continue to |           |
| be encouraged. It allows panelists to be better prepared |           |
| for the panel itself and helps to give enough time for   |           |
| all the work to be done correctly in the short period of |           |
| time allotted to a given panel. We suspect that the      |           |
| interactive panel system is improving the quality of the |           |
| reviews and providing an incentive for panelists to      |           |
| finish their reviews in advance of the meeting to avoid  |           |
| embarrassment. The NSF is to be commended for having the |           |
| insight to build such a high quality system as           |           |
| Fastlane..                                               |           |
+----------------------------------------------------------+-----------+
| Is the time to decision appropriate?                     |           |
|                                                          |           |
| Comments: We believe that the interactive panel system   |           |
| should encourages panelists to write higher quality      |           |
| summaries. Certainly the process of modifying and        |           |
| signing off on the summaries has become far more         |           |
| effective with the messaging system that is part of the  |           |
| interactive panel system. Our examination of the sample  |           |
| jackets reveals, however, that most summaries were less  |           |
| informative than the individual reviews and did not      |           |
| always sufficiently explain, for example, why a proposal |           |
| is rejected. If more time could be devoted to writing    |           |
| summaries during the panel meeting, they might improve.  |           |
| The program manager should also be careful in picking    |           |
| who writes the summary for the proposal. Perhaps someone |           |
| not participating in the discussion of a particular      |           |
| proposal can do the best job as a scribe of the          |           |
| discussion even though he/she is less expert on that     |           |
| particular proposal.                                     |           |
+----------------------------------------------------------+-----------+
| Discuss issues identified by the COV concerning the      |           |
| quality and effectiveness of the program's use of merit  |           |
| review procedures:                                       |           |
|                                                          |           |
| The program's use of merit review procedures assures     |           |
| high quality and effectiveness. But again, we recommend  |           |
| more use of ad hoc reviews to help: (i) balance reviews  |           |
| of panels that may not have enough members with          |           |
| expertise in the proposal topics, or (ii) avoid the bias |           |
| by an outspoken influential member of the panel. Panels  |           |
| are useful in allowing a program officer to quickly      |           |
| decide on controversial proposals and even to override   |           |
| reviews by observing reviewers biases. It is important   |           |
| in this case for the Program Director to take the        |           |
| appropriate action by either seeking more reviews or     |           |
| make a decision based on his/her evaluation.             |           |
|                                                          |           |
| We were unable to determine how often the program        |           |
| manager might have overruled the decisions of the panel. |           |
| It would be helpful if the front matter of the jacket    |           |
| had had a list of the other proposals from that panel    |           |
| that were funded (it has the rankings but not the actual |           |
| funding decisions).                                      |           |
|                                                          |           |
| We discussed whether it might be possible to identify    |           |
| those proposals that would require substantial           |           |
| discussion in advance of the meeting. This would allow   |           |
| panelists to re-read the appropriate proposals in        |           |
| advance of the meeting. This time for advance            |           |
| preparation might mitigate the ability of some panelists |           |
| to take over the meeting as those who are less           |           |
| spontaneously articulate could prepare their statements  |           |
| in advance. One way to help flag problem cases would be  |           |
| to solicit one or two ad hoc mail reviews from           |           |
| appropriate experts who are asked to reply in a timely   |           |
| manner before the panel meeting. If the mail reviews are |           |
| substantially different from the panel, then this would  |           |
| alert the Program Director to a problem case in need of  |           |
| special treatment.                                       |           |
|                                                          |           |
| The workload on the Program Director is rather heavy,    |           |
| especially considering the additional load imposed by    |           |
| the ITR program without corresponding increase in        |           |
| personnel.                                               |           |
+----------------------------------------------------------+-----------+

**A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers.**

a.  The program is [successful]{.underline} when reviewers address the
    elements of both generic review criteria appropriate to the proposal
    at hand and when program officers take the information provided into
    account in their decisions on awards.

b\. Identify possible reasons for dissatisfaction with NSF's merit
review system.

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       |           |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments: Not much attention appeared to be paid to      |           |
| broader impacts in the jackets that we reviewed. These   |           |
| proposals all dated from before the new rule emphasizing |           |
| the importance of broader impact. Intellectual merit was |           |
| appropriately addressed in the jackets that we reviewed. |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: The panel summaries were more likely to        |           |
| address both criteria, perhaps because of instructions   |           |
| provided at the panel meeting.                           |           |
+----------------------------------------------------------+-----------+
| Have the review analyses (Form 7s) addressed whether the |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: While PI's often have definitive ideas about   |           |
| intellectual merit criteria, some are not sure what to   |           |
| state as the relevant components of broader impact of    |           |
| the proposed research. More explicit instructions by the |           |
| NSF would be helpful**.**                                |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system.               |           |
|                                                          |           |
| The panel summaries tended to be somewhat sparse in      |           |
| information. In the case of acceptance the reviews often |           |
| tended to refer to the strong reputations of the         |           |
| particular PIS. In the case of rejection the reviews     |           |
| tended to focus on the shortcomings of the application.  |           |
| However at some level the shortcomings mentioned         |           |
| implicitly resulted from observed problems with          |           |
| intellectual content or lack of impact.                  |           |
|                                                          |           |
| This comment is the same as one made in the previous     |           |
| three--year review. The program has been successful in   |           |
| recruiting reviewers that properly address the review    |           |
| criteria. However, we see a threat to the integrity of   |           |
| the review process as the number of proposals increase.  |           |
| With the ITR program doubling the number of proposals,   |           |
| and more than doubling of workload, the same number of   |           |
| NSF staff cannot manage review processes at the same     |           |
| high level of quality achieved in the past.              |           |
+----------------------------------------------------------+-----------+

**A.3 Questions concerning the selection of reviewers.**

+---------------------------------------------------------+------------+
| **Selection of Reviewers**                              | **YES,     |
|                                                         | NO,**      |
|                                                         |            |
|                                                         | **DATA NOT |
|                                                         | AVAILABLE, |
|                                                         | or NOT     |
|                                                         | AP         |
|                                                         | PLICABLE** |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       |            |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments: The panels assembled seemed to not always     |            |
| have members with the appropriate expertise for the     |            |
| proposals under review. More mail/ad hoc reviewers      |            |
| might help to address this problem. For example, oOne   |            |
| of the jackets that we reviewed had no panelists who    |            |
| were in the area of the proposal (animation within      |            |
| graphics).                                              |            |
|                                                         |            |
| .                                                       |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            |            |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments: Given the relatively small size of the panel, |            |
| the program manager must be very careful to assemble a  |            |
| group of people with appropriate expertise for a given  |            |
| set of panels. Graphics in particular is a broad area   |            |
| that is very difficult to cover with a panel of only    |            |
| 7-8 people. We expect that the number of submitted      |            |
| proposals in graphics is likely to increase             |            |
| substantially over the next few years with all of the   |            |
| strong young people who have recently entered academia  |            |
| and the size of the panel will have to grow to match.   |            |
| Submissions to the premiere graphics conference,        |            |
| SIGGRAPH, have been increasing at a rate of 20% for     |            |
| each of the past several years and it is reasonable to  |            |
| expect a similar increase in submitted proposals.       |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    |            |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments: There does not seem to be an effective way    |            |
| for resolving conflicts of interest, during the process |            |
| of proposal rankings, at the end of the panel review.   |            |
| At that point in the process, the panel must rank       |            |
| proposals relative to each other, and proposals are     |            |
| discussed many times making it impossible for those     |            |
| conflicted to leave the room each time the proposals    |            |
| that they are conflicted with is mentioned. We suggest  |            |
| that the program manager should try to address this     |            |
| problem by doing as much as he can to identify          |            |
| conflicts during the panel selection process (without,  |            |
| of course, sacrificing the quality of the expertise on  |            |
| the panel).                                             |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      |            |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments: One concern identified regarding the          |            |
| selection of reviews is that: Given a collection of     |            |
| proposals covering a number of sub-areas in numerics    |            |
| (for example), how does a Program Manager assure equal  |            |
| representation of expertise on the panel in each area   |            |
| for fair evaluation?                                    |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
|                                                         |            |
| The subcommittee noted that there appeared to be more   |            |
| reviewers from the east coast than from the west coast  |            |
| or from the mid-west however we have no way of knowing  |            |
| whether this is just a reflection of the density of     |            |
| schools on the east coast or real geographic imbalance. |            |
|                                                         |            |
| There was no information on cases where the program     |            |
| directors needed to rely on ad hoc reviewers after      |            |
| panel decisions were made. Did this happen?             |            |
+---------------------------------------------------------+------------+

**A.4 Questions concerning the resulting portfolio of awards under
review.**

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     |               |
| projects supported by the program.                   |               |
|                                                      |               |
| Comments: The average size of the awards is small    |               |
| given the cost of faculty salaries and student       |               |
| tuition and stipends. Not all the individual awards  |               |
| are sufficient to cover one year of student support  |               |
| and one month of faculty salary.                     |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  |               |
| scope of the projects?                               |               |
|                                                      |               |
| Comments: The proposals do not appear to be          |               |
| particularly high risk. However, the ITR program was |               |
| in place during this period and presumably the       |               |
| proposals funded there were higher risk so we do not |               |
| see this as a problem until the end of the ITR.      |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| High Risk Proposals?                                 |               |
|                                                      |               |
| Comments: No significantly interdisciplinary         |               |
| proposals were reviewed. As would be expected with a |               |
| topic-based system such as the NSF directorate, it   |               |
| is easy for interdisciplinary proposals to slip      |               |
| through the cracks with each potential discipline    |               |
| feeling that the proposal isn't "central" enough to  |               |
| warrant spending their limited funds on it.          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of: Multidisciplinary Proposals?             |               |
|                                                      |               |
| Comments: No center or group proposals were          |               |
| reviewed. The proposals reviewed were CAREER,        |               |
| individual, and SGER.                                |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Innovative Proposals?                                |               |
|                                                      |               |
| Comments: While the number of funded CAREER          |               |
| proposals in numerical computing is very low, it     |               |
| should be noted that very few CAREER proposals were  |               |
| submitted in this area. There were no CAREER         |               |
| proposals from symbolic computation.                 |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Funding for centers, groups and awards to            |               |
| individuals?                                         |               |
|                                                      |               |
| Comments: In graphics, there is a very strong crop   |               |
| of junior faculty at the moment and the size of the  |               |
| CAREER program must be increased correspondingly. In |               |
| 2002, 15 CAREER grants were submitted and only two   |               |
| were funded. This low acceptance rate is potentially |               |
| very damaging to the otherwise promising careers of  |               |
| these junior faculty.                                |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Awards to new investigators?                         |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Geographical distribution of Principal               |               |
| Investigators?                                       |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Institutional types?                                 |               |
|                                                      |               |
| Comments: T here is a good mix of awards to top      |               |
| research universities and primarily teaching         |               |
| institutions. The percentage of such funded          |               |
| proposals is also quite appropriate.                 |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Projects that integrate research and education?      |               |
|                                                      |               |
| Comments: Other than the CAREER grants, the projects |               |
| that we reviewed did not often particularly          |               |
| emphasize the integration of research and education. |               |
| Some proposals in symbolic computation did discuss   |               |
| such integration, however the GSG program has funded |               |
| an impressive collection of innovative disciplinary  |               |
| proposals.                                           |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance:                                             |               |
|                                                      |               |
| Across disciplines and subdisciplines of the         |               |
| activity and of emerging opportunities?              |               |
|                                                      |               |
| Comments: There seemed an appropriate balance among: |               |
| (i) numerical & optimization, (ii) geometric &       |               |
| symbolic, and (iii) graphics. Lately however some    |               |
| members of the committee felt that more emphasis     |               |
| should be given to proposals with a strong           |               |
| dependence on graphics and geometry. Perhaps         |               |
| proposal pressure should be considered within the    |               |
| GSG program as well as between programs.             |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          |               |
| participation of underrepresented groups?            |               |
|                                                      |               |
| Comments: The number of proposals submitted by       |               |
| underrepresented groups is rather low, however.      |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      |               |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments: The successful proposals all tended to be  |               |
| incremental in what they hope to accomplish.         |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
+------------------------------------------------------+---------------+

**A.5 Management of the program under review**

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments: The GSG program has been managed efficiently; the division  |
| director and the program managers should be commended for their       |
| efforts.                                                              |
|                                                                       |
| It should be noted, however, that the program has had three program   |
| managers during the time frame that we are considering. The program   |
| needs greater continuity in program managers.                         |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments: The program has not responded to the growing number of      |
| faculty in graphics and the resulting increase in submitted proposals |
| (particularly in the CAREER program).                                 |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments: The program planning and prioritization process (internal   |
| and external) that guided the development of the portfolio under      |
| review was not visible to the COV.                                    |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| The subcommittee was not unanimous in its belief, that having         |
| graphics and computational geometry merged with numeric and symbolic  |
| computing is the way this program would foster in future years, nor   |
| be expertly managed. It was also thought extremely difficult for a    |
| program manager to be sufficiently expert in these varied sub-areas   |
| to pick reasonable panels.                                            |
|                                                                       |
| The GSG program has been responsive to emerging areas in graphics,    |
| geometric, numeric and symbolic computing, however the task appears   |
| increasingly daunting for future years.                               |
+-----------------------------------------------------------------------+

##### B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.**

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

+-----------------------------------------------------------------------+
| **[B.1 NSF OUTCOME GOAL for PEOPLE:]{.underline}** **Developing "a    |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| Comments: The program is [successful]{.underline} when participants   |
| in NSF activities experience world-class professional practices in    |
| research and education, using modern technologies and incorporating   |
| international points of reference; when academia, government,         |
| business, and industry recognize their quality; and when the science  |
| and engineering workforce shows increased participation of            |
| underrepresented groups.                                              |
+-----------------------------------------------------------------------+
| **[B.2 NSF OUTCOME GOAL for IDEAS:]{.underline} Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| **Comments:** Performance is [successful]{.underline} when NSF awards |
| lead to important discoveries; new knowledge and techniques, both     |
| expected and unexpected, within and across traditional disciplinary   |
| boundaries; and identification of high potential links across these   |
| boundaries.                                                           |
|                                                                       |
| The GSG program is concerned with the computational foundations of    |
| science and engineering. Mathematics and computations are needed in   |
| scientific and engineering work, such as modeling natural phenomena   |
| or designing artifacts. Traditionally, a scientist or engineer solved |
| a problem by setting up equations among the entities involved         |
| (assuming simple, idealized geometries if geometric configuration had |
| any role in the problem), then performing mathematical formula        |
| manipulations by hand, until the solution could be obtained by        |
| numerical evaluation. All the mathematical derivations involving      |
| algebra, analysis, geometry, logic, etc. were done manually as        |
| problem preprocessing, and machine computations were limited to       |
| numerical work. To cope with the growing complexity of scientific     |
| research and engineering design problems, techniques are needed for   |
| performing very large mathematical calculations symbolically by       |
| machine. Scientific and engineering work is also becoming more        |
| computational because, increasingly, computer simulation is replacing |
| physical experimentation and the construction and testing of          |
| prototypes. For realistic, detailed simulations, the computational    |
| arsenal available to research scientists and engineers has to be      |
| greatly enhanced. In general, algorithmic and computational           |
| techniques have to be coupled with methods of analysis for all        |
| branches of mathematics used in science and engineering. For          |
| engineering design work, computational methods for optimization are   |
| also critical.                                                        |
|                                                                       |
| **Subareas**                                                          |
|                                                                       |
| The goal of the NSG program is the advancement of the computational   |
| infrastructure for science and engineering via research in topics in  |
| which computational techniques are coupled with thematical methods of |
| analysis. The particular areas change slightly from year to year, but |
| currently include the following, roughly in order of largest to       |
| smallest, as determined by proposal pressure.                         |
|                                                                       |
| **Computer Graphics**                                                 |
|                                                                       |
| -   Motion transformations for animation;                             |
|                                                                       |
| -   Multi-resolution methods, mainly focusing on subdivision schemes, |
|     wavelet methods, and ideal data representation in higher          |
|     dimensions;                                                       |
|                                                                       |
| -   Investigate extraction of general, three-dimensional human and    |
|     animal models and their motions from digitized video;             |
|                                                                       |
| -   Image-based rendering.                                            |
|                                                                       |
| **Computational Geometry**                                            |
|                                                                       |
| -   Core problems (hulls, intersections, visibility, \...);           |
|                                                                       |
| -   Parallel algorithms;                                              |
|                                                                       |
| -   Applications to solid modeling, graphics, theorem proving,        |
|                                                                       |
| -   Biology, robotics, manufacturing, metrology.                      |
+-----------------------------------------------------------------------+
| **Numerical Methods**                                                 |
|                                                                       |
| -   Matrix computation (sparse and structured);                       |
|                                                                       |
| -   Nonlinear equations;                                              |
|                                                                       |
| -   Pdes (finite-element and adaptive methods);                       |
|                                                                       |
| -   Analytical complexity;                                            |
|                                                                       |
| -   Supercomputing, parallel computing;                               |
|                                                                       |
| -   Scientific applications, applied maths;                           |
|                                                                       |
| -   Software, libraries.                                              |
|                                                                       |
| **Symbolic & Algebraic Computation**                                  |
|                                                                       |
| -   Computational algebra & analysis;                                 |
|                                                                       |
| -   Computational polynomial algebra, groebner bases;                 |
|                                                                       |
| -   Computational algebraic geometry;                                 |
|                                                                       |
| -   Computational number theory;                                      |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Integration, de solution;                                         |
|                                                                       |
| -   Complexity of algebraic algorithms;                               |
|                                                                       |
| -   Numeric-symbolic methods;                                         |
|                                                                       |
| -   Scientific applications;                                          |
|                                                                       |
| -   Systems and software.                                             |
|                                                                       |
| **Optimization & Mathematical Programming**                           |
|                                                                       |
| -   Numerical optimization;                                           |
|                                                                       |
| -   Linear programming, interior-point methods, complexity and robust |
|     algorithms;                                                       |
|                                                                       |
| -   Nonlinear problems; network optimization;                         |
|                                                                       |
| -   Large-scale problems.                                             |
|                                                                       |
| **Theorem Proving**                                                   |
|                                                                       |
| -   Resolution-based;                                                 |
|                                                                       |
| -   Non-resolution, graphical methods;                                |
|                                                                       |
| -   Higher-order theorem proving;                                     |
|                                                                       |
| -   Rewrite rules;                                                    |
|                                                                       |
| -   Applications to algebra, analysis, geometry;                      |
|                                                                       |
| -   Applications to symbolic computations.                            |
|                                                                       |
| **Problem Solving Environments**                                      |
|                                                                       |
| -   Productivity-enhancing tools.                                     |
|                                                                       |
| The GSG program supports several special forms of awards to support a |
| diverse scientific workforce, which are detailed below. In addition,  |
| the Program Director also makes specific suggestions, and encourages  |
| his PIs, to submit more requests. Furthermore, GSG cost-shares        |
| proposals with several other NSF divisions, such as International[^7] |
| and DMS[^8]. In 2001, 23 PIs currently managed or funded by GSG       |
| (under its previous name NSG) were female. These included for example |
| Amato[^9], Amenta[^10], Arkin[^11], Blumenthal[^12], Bonacina[^13],   |
| Briggs[^14], Dorsey[^15], Grimm[^16], and Hodgins[^17].               |
+-----------------------------------------------------------------------+

Specific examples of new research areas fostered by each of the subareas
of GCG:

**Animation work of Zoran Popovic**

Zoran Popovic has several grants through CSG. He is one of the most
promising young researchers in computer graphics and will be up for
tenure soon. His CSG funding has resulted in three SIGGRAPH papers in
each of 2002 and 2003. These papers have spanned a wide range of topics
ranging from the use of optimization to create motion for computer
animation to data-driven techniques for scanning and modeling human
bodies.

## **Bit complexity of the determinant of an integral matix**

##### Erich Kaltofen[^18] (NCSU) and Gilles Villard (ENS-Lyon) have discovered a new algorithm that computes the determinant of an integer matrix of dimensions n by n whose entries have no more than L bits in (n^1+1/3^ L)^1+o(1)^ fixed precision (bit) operations. The algorithm uses cubic matrix operations, but is asymptotically faster than the standard method based on Strassen-Coppersmith/Winograd matrix multiplication and Chinese remaindering, which would yield (n^3.3755^ L)^1+o(1)^ bit operations. In fact, with fast matrix multiplication the exponent of this methods drops to 2.698. Their algorithm combines methods for symbolic computation (Chinese remaindering, baby steps-giant steps), coding theory (the Berlekamp Massey algorithms), numerical computation (block Lancsoz), and control theory (multivariable realizations).

##### 

##### The significance in this result is the fact that the interplay between the algebraic structure of the determinant and the bits of the intermediately computed numbers can lead to a dramatic reduction in the cost of computing the determinant, that without using the asymptotically fast but impractical matrix multiplication algorithms. It is expected that further study of the phenomenon of arithmetic vs. bit manipulation will yield faster algorithms for many problems in symbolic computation.

##### 

**Multiresolution Dynamic Models**

This is work by C Bajaj[^19] at UT Austin.

##### Multi-resolution techniques and models have been shown to be effective for the display and transmission of very large static geometric objects. Dynamic environments (e.g. time-varying simulations, 3D games) with internally deforming objects pose similar challenges in terms of time and space and need the development of similar solutions. This past year, they have developed the T-DAG, an adaptive multi-resolution representation for dynamic meshes with arbitrary deformations including attribute, position, connectivity and topology changes. They also have developed an on-line algorithm for constructing the T-DAG, enabling the traversal and use of the multi-resolution model for partial playback while it is still being constructed. 

##### 

##### Further information is available from the project web pages <http://www.ices.utexas.edu/CCV>.

## 

## Computing with Shapes: Reconstruction and Decimation

## PI: Tamal Dey, Ohio State University, 2000\--2003.

Sampling a geometric object with laser scanners and then producing a
computer model out of this sample point cloud is becoming increasingly
popular in CAD designs, computer imaging and many other applications. We
devised new methods for this modeling paradigm and designed a software
called cocone for this problem. The software has been released to the
public. Researchers from various academic and industrial labs have
downloaded the software. We have published our results in reputed
journals and conference proceedings in the field of computational
geometry and visualization.

More about the results can be obtained via the web link
http://www.cis.ohio-state.edu/\~tamaldey and following other links from
it.

**The Simplex Method is Provably Polynomial**

During the last two years Shang-Hua Teng[^20] (U of Illinois at Urbana
Champagne), together with Dan Spielman (MIT), solved a long-outstanding
open question in Mathematical Programming, Optimization, and Theoretical
Computer Science. They proved that the Simplex Method for Linear
Programming usually takes polynomial number of steps. In the process,
they developed a new algorithm-analysis framework that we call smoothed
analysis that can help explain the success of many algorithms and
heuristics that the traditional algorithm-analysis frameworks, such as
the worse-case and average-case analysis, cannot. The simplex algorithm
is the classic example of an algorithm that is known to perform well in
practice but which takes exponential time in the worst case. It has been
an active subject for both mathematical and experimental studies for
more than 50 years.

The Analysis of Algorithms community has been challenged by the
existence of remarkable algorithms that are known by scientists and
engineers to work well in practice, but whose theoretical analyses
negative or inconclusive. The root of this problem is that algorithms
are usually analyzed in one of two ways: by worst-case or average-case
analysis. Worst-case analysis can improperly suggest that an algorithm
will perform poorly by examining its performance under incredibly
contrived circumstances that may never occur in practice. On the other
hand, while many algorithms perform unusually well on random inputs
considered in average-case analysis, randomly generated inputs often
bear little resemblance to those actually encountered in practice.

They propose an analysis that called smoothed analysis that can help
explain the success of many algorithms that both worst-case and average
case cannot. In smoothed analysis, we measure the performance of an
algorithm under slight random perturbations of arbitrary inputs. In
particular, they consider absolute and relative Gaussian perturbations
of inputs to algorithms that take real and complex inputs, and they
measure the running time of algorithms in terms of the input size and
the variance of the Gaussian perturbations.

They show that the simplex algorithm has polynomial smoothed complexity.
The simplex algorithm is the classic example of an algorithm that is
known to perform well in practice but which takes exponential time in
the worst case. It was introduced by Dantzig in mid-1940th during the
Second World War, and remains widely used today various industrial and
military optimizations. In the late 1970\'s and early 1980\'s the
simplex algorithm was shown to converge in expected polynomial time on
various distributions of random inputs by researchers including
Borgwardt and Smale. However, the last 20 years of research in
probability, combinatorics and numerical analysis have taught us that
these random instances have very special properties that one should not
expect to find in practice.

## **High performance computational techniques for solving sparse linear systems on parallel and distributed computers**

The NSF awards to Jun Zhang[^21], U of Kentucky, support research and
development of high performance computational techniques for solving
sparse linear systems on parallel and distributed computers with
scientific, engineering, and industrial applications. Under the support
of these NSF grants, the Laboratory for High Performance Scientific
Computing and Computer Simulation (the HiPSCCS Lab) at the University of
Kentucky has developed new robust iterative methods for solving
difficult sparse matrices on parallel and distributed computers.

Zhang's new techniques have caught the attention of some application
scientists. Research and development engineers at the General Motors
Company expressed interest in using the parallel linear system solvers
developed at the HiPSCCS Lab in their car engine combustion code to
speed-up their large scale computer simulation in designing new cars.
Researchers at the Japanese Research Organization for Information
Science & Technology (RIST) is interested in utilizing our high
performance linear system solvers in conducting very large scale
computer simulation of solid Earth. Their goal is to attempt more
accurate predication of earthquakes. Thus, results from the NSF funded
research projects have the potential to make people\'s life better by
designing new and safer cars, and to save human lives by predicting
natural disasters more accurately.

The NSF funded research projects enable the HiPSCCS Lab to support and
collaborate with women and members from underrepresented groups in
cutting-edge research on high performance scientific computing. About
50% students and postdocs trained at the HiPSCCS Lab are women. In
addition, Professor Jennifer J. Zhao (woman) from the University of
Michigan at Dearborn visited the HiPSCCS Lab on an NSF Research
Opportunity Award. Dr. Jules Kouatchou (minority) from Morgan State
University has been collaborating on one of the NSF funded research
projects. The NSF funded research projects provide training opportunity
for graduate students, postdocs, and visitors, to study and conduct
research in high performance scientific computing and computer
simulation and educate skilled information technology workers for the
U.S. economy.

> GSG, through its support for meetings such as ECCAD plays an important
> role in both bringing new graduate students into the symbolic algebra
> research community. In addition, through its support of such workshops
> as IAMC, GSG plays a very important role in bringing symbolic
> computation researchers together to help define and clarify the role
> of new technologies in both research and education.

**B.3 OUTCOME GOAL for TOOLS: Providing "broadly accessible,
state-of-the-art and shared research and education tools."**

Comments: The program is [successful]{.underline} if NSF awards lead to
the development, adoption, adaptation, and implementation of effective
models, products, and practices that address the needs of all students;
well-trained teachers who implement standards-based approaches in their
classrooms; and improved student performance in participating schools
and districts.

#####  **C. OTHER TOPICS**

**C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.**

We believe that NSF in general and CCR in particular has to designate a
specific budget for CAREER awards with well-advertised expected number
of overall CAREER grants to be awarded in a given year and the average
size of a grant. This will relay the message to CS and Math departments
throughout the country regarding the competitive nature of these grants
and that not every junior faculty member is entitled to a CAREER grant.

**C.2 Please provide comments as appropriate on the program's
performance in meeting program-specific goals and objectives that are
not covered by the above questions.**

It seems that the NSF intends to encourage high risk/high payoff
proposals. Unfortunately, we did not find any such example in the
proposals funded by GSG. Panelists are often conservative and averse to
taking risks. Program Directors need to assert leadership by identifying
high risk/high payoff proposals and rendering appropriate judgment.

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

More administrative funds are needed to increase support for Program
Directors to enable them to handle the tremendous proposal load.

**C.4 Please provide comments on any other issues the COV feels are
relevant.**

For every 10 proposals it appears that 8 deserve funding, that 5-6 very
much deserve

funding but that at the end only 3 actually get funding. So of course
the proposals that

are actually supported in the GSG subdivision are outstanding. However
it is also the

case that there are a significant number of excellent proposals that
never get funded.

Point C.3 notes that this will get to be more of a problem in the
future. Point C.2 is likely

a direct outcome of the difficulty in obtaining any sort of funding.

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

There is very little mention of significant mathematical software in the
proposals that

were looked at. One can imagine that such software would help with
obtaining a

broaderimpact for many research proposals. We do note, however, that the
amount of

funding awarded for an average successful proposal severely limits the
ability to spend

time creating such software for wide use.

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the Graphics, Symbolic and Geometric Computation (GSG)

\[Chandrajit Bajaj\]

Chair

# Computer Systems Architecture (CSA)

##### **FY 2003 REPORT TEMPLATE FOR**

#####  **NSF COMMITTEES OF VISITORS (COVs)**

  -----------------------------------------------------------------------
  **Date of COV: June 5-6, 2003**

  **Program/Cluster: Computer Systems Architecture**

  **Division: CCR**

  **Directorate: CISE**

  **Number of actions reviewed by COV**[^22]**:\
  Awards: 9 Declinations: 9 Other: 0**

  **Total number of actions within Program/Cluster/Division during period
  being reviewed by COV**[^23]**: 355 Awards: 133 Declinations: 215
  Other:7**

  **Manner in which reviewed actions were selected:\
  **
  -----------------------------------------------------------------------

**A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

**A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures..**

+----------------------------------------------------------+-----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES** | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Is the review mechanism appropriate? (panels, ad hoc     |           |
| reviews, site visits)                                    |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Is the review process efficient and effective?           |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Are reviews consistent with priorities and criteria      |           |
| stated in the program's solicitations, announcements,    |           |
| and guidelines?                                          |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Do the individual reviews (either mail or panel) provide |           |
| sufficient information for the principal investigator(s) |           |
| to understand the basis for the reviewer's               |           |
| recommendation?                                          |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Do the panel summaries provide sufficient information    |           |
| for the principal investigator(s) to understand the      |           |
| basis for the panel recommendation?\                     |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Is the documentation for recommendations complete, and   |           |
| does the program officer provide sufficient information  |           |
| and justification for her/his recommendation?            |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Is the time to decision appropriate?                     |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Discuss issues identified by the COV concerning the      |           |
| quality and effectiveness of the program's use of merit  |           |
| review procedures:                                       |           |
|                                                          |           |
| We feel that the current panel system yields timely      |           |
| reviews, and that the panelists selected are qualified   |           |
| in their fields.                                         |           |
|                                                          |           |
| However, panels may tend to emphasize the opinion of one |           |
| panelist who happens to be the expert in the domain of   |           |
| the proposal under review. One panelist may influence    |           |
| the others. One possibility may be to increase the       |           |
| number (and importance given to) ad hoc reviews.         |           |
|                                                          |           |
| The workload given to each panelist appears to be too    |           |
| high. It should be reduced so as to increase the detail  |           |
| of the reviews, and reach what is usually obtained for a |           |
| high quality conference. More detailed reviews would be  |           |
| desirable and would provide better feedback to the PIs.  |           |
| This could be achieved by lowering the panelists'        |           |
| workload and/or increasing the number of external        |           |
| reviews.                                                 |           |
|                                                          |           |
| As for the final recommendation, we felt that a          |           |
| carefully drafted "synthesis" of the evaluation should   |           |
| be provided by the panel in addition to the "weaknesses" |           |
| and "strong points." Otherwise, this renders the panel's |           |
| recommendation too sketchy.                              |           |
|                                                          |           |
| No other issues were identified by this COV concerning   |           |
| the quality and effectiveness of the program's use of    |           |
| merit review procedures.                                 |           |
+----------------------------------------------------------+-----------+

**A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers.**

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       |           |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Have the review analyses (Form 7s) addressed whether the |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system.               |           |
|                                                          |           |
| We felt that the NSF merit review criteria (intellectual |           |
| merit and broader impact) have been well-implemented by  |           |
| the panelists and the program director.                  |           |
|                                                          |           |
| By and large, the individual reviews (both mail and      |           |
| panel) addressed whether proposals contribute to both    |           |
| merit review criteria. However, as mentioned above, the  |           |
| panel summary reviews are usually too sketchy to infer   |           |
| this information. The review analyses (Form 7s) have     |           |
| usually addressed whether the proposal contributes to    |           |
| both merit review criteria.                              |           |
|                                                          |           |
| Overall, this subcommittee recommends that high-risk     |           |
| research and "established" research should both be given |           |
| an equal chance. This means that reviewers should not    |           |
| downgrade a proposal for lack of preliminary results, if |           |
| the proposal is otherwise promising.                     |           |
+----------------------------------------------------------+-----------+

**A.3 Questions concerning the selection of reviewers.**

+---------------------------------------------------------+------------+
| **Selection of Reviewers**                              | **YES,     |
|                                                         | NO,**      |
|                                                         |            |
|                                                         | **DATA NOT |
|                                                         | AVAILABLE, |
|                                                         | or NOT     |
|                                                         | AP         |
|                                                         | PLICABLE** |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       |            |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments: See Comments Below                            |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            |            |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments: See Comments Below                            |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    |            |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments: See Comments Below                            |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      |            |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments: See Comments Below                            |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
|                                                         |            |
| The subcommittee would like to reiterate the following  |            |
| observations: An additional number of outside reviewers |            |
| would be desirable. This would make for a better        |            |
| balance against this opinion of a panel which may be    |            |
| biased by its own dynamics. However, it should be noted |            |
| that all panelists and reviewers (no exceptions noted)  |            |
| had excellent background and expertise, but that their  |            |
| effectiveness may have been hampered by excessive       |            |
| workload.                                               |            |
|                                                         |            |
| The selection of reviewers was a careful balance of     |            |
| geographical location, representation of all groups,    |            |
| and topics. Conflicts of interest were properly         |            |
| detected and handled.                                   |            |
+---------------------------------------------------------+------------+

**A.4 Questions concerning the resulting portfolio of awards under
review.**

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     |               |
| projects supported by the program.                   |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  |               |
| scope of the projects?                               |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| High Risk Proposals?                                 |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of: Multidisciplinary Proposals?             |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Innovative Proposals?                                |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Funding for centers, groups and awards to            |               |
| individuals?                                         |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Awards to new investigators?                         |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Geographical distribution of Principal               |               |
| Investigators?                                       |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Institutional types?                                 |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Projects that integrate research and education?      |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance:                                             |               |
|                                                      |               |
| Across disciplines and subdisciplines of the         |               |
| activity and of emerging opportunities?              |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          |               |
| participation of underrepresented groups?            |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      |               |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments: See Comments Below                         |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
|                                                      |               |
| The subcommittee found that the overall quality of   |               |
| the research and/or education projects supported by  |               |
| the program was high. However, given that many       |               |
| projects are reduced in budget, with a commensurate  |               |
| reduction in the scope of work, it appears that the  |               |
| resources (total budget of CSA) should be increased. |               |
| Also, ambitious, risk-taking projects which require  |               |
| more resources (e.g., for prototyping, or for the    |               |
| purpose of building interdisciplinary teams), should |               |
| be encouraged. In terms of individual topics, the    |               |
| subcommittee members pointed to the fact that Memory |               |
| Systems awards have gone down from 6 down to 1 (2000 |               |
| to 2002). Given that the memory wall problem is an   |               |
| increasingly important architectural design issue,   |               |
| it could be a concern if this trend were to          |               |
| continue. Please refer to part C of this document    |               |
| for a more detailed discussion.                      |               |
|                                                      |               |
| The program had an appropriate balance of new        |               |
| investigators, geographical distribution, and        |               |
| diversity of institutional types.                    |               |
+------------------------------------------------------+---------------+

**A.5 Management of the program under review**

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments: See Comments Below                                          |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments: See Comments Below                                          |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments: See Comments Below                                          |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| In terms of the management of the program, the subcommittee remarked  |
| the following:                                                        |
|                                                                       |
| -   Proposal Review and Award selection: in general, judicious        |
|     choices have been made under uncertain conditions. They have been |
|     encouraging to innovative projects, and balanced and fair. The    |
|     subcommittee appreciates the performance of the program managers. |
|                                                                       |
| -   Budget: too many "competitive" proposals had to be declined (only |
|     22% funded) due to budget constraints.                            |
|                                                                       |
| -   50% of budget "borrowed" from the following year to fund projects |
|     in the current year may be too high.                              |
|                                                                       |
| The program is encouraged to continue its involvement in new emerging |
| research areas such as the Nanoscale Science and Engineering Programs |
| and Initiative.                                                       |
|                                                                       |
| Workshops such as the NSF Workshop on Critical Issues in Computer     |
| Architecture have been instrumental in providing a vision and general |
| direction to the program. This is an effective tool for community     |
| feedback.                                                             |
+-----------------------------------------------------------------------+

##### B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.**

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

+-----------------------------------------------------------------------+
| **[B.1 NSF OUTCOME GOAL for PEOPLE:]{.underline}** **Developing "a    |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| **Comments**: The CSA program encourages the development of PhD       |
| students in the important area of computer architecture and systems.  |
| It also provides researchers access to state-of-the-art equipment and |
| other research infrastructures, which improve productivity in         |
| producing research results. The PI's supported by the CSA program     |
| have received numerous awards, such the Alfred P. Sloan Research      |
| Fellowships (To Profs. Sarita Adve, Doug Burger and Steve Keckler). A |
| PI has become an IEEE fellow (Prof. Sachin Sapatekhar),. PIs have     |
| also received best paper awards in international conferences and have |
| acquired many patents as a result of their research. Also, the        |
| subcommittee found that CSA program has increased the REU (Research   |
| Experience for Undergraduate students) awards to 8 in 2002, which we  |
| feel is a positive encouragement for the participation of             |
| undergraduate students in research activities.                        |
+-----------------------------------------------------------------------+
| **[B.2 NSF OUTCOME GOAL for IDEAS:]{.underline} Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| **Comments:** This program is successful in this regard. We have      |
| identified the following relevant information which substantiates     |
| this NSF outcome:                                                     |
|                                                                       |
| -   A patent application entitled \"Method and System for             |
|     Synthesizing a Circuit Representation into a New Circuit          |
|     Representation Having Greater Unateness,\" was filed by the       |
|     University of Michigan with the U.S. Patent and Trademarks Office |
|     in August 2001. The patent co-inventors are Hyungwon Kim (a       |
|     former Ph.D. student at Michigan, now working at Broadcom Inc.)   |
|     and J. P. Hayes.. The invention covers a method to automatically  |
|     decompose digital circuits of arbitrary structure into small      |
|     blocks that exhibit the unateness property. The resulting         |
|     block-structured designs have several advantages including high   |
|     performance, low area, and small test sets.                       |
|                                                                       |
|     Dr. Hayes is a co-PI on the NSG Grant CCR-0208958, "Sequential    |
|     Architectures for Quantum Computation"                            |
|                                                                       |
| -   Dr. Wei Hsu has filed the patent, \"Method of Efficient Dynamic   |
|     Data Cache Prefetch Insertion", in 2002.                          |
|                                                                       |
|     Dr. Hsu is supported by NSF grant CCR 0105574, "ADORE \-- A       |
|     Framework for Adaptive Object Code Re-optimization"               |
|                                                                       |
| -   Four U.S. patent applications have been filed on Dr. Ruby Lee\'s  |
|     work on innovative permutation instructions for fast software     |
|     cryptography.  Current processors take hundreds of cycles to      |
|     achieve any one of the 64! permutations possible when permuting   |
|     the bits of a 64-bit plaintext block,  a very useful              |
|     \"diffusion\" operation in symmetric-key cryptography. Dr. Lee's  |
|     research showed how to do this with at most 6 instructions in any |
|     processor, and in 1 cycle on a 4-way superscalar processor.       |
|                                                                       |
| > Dr. Lee's research is supported in part by NSF grant CCR-0105677,   |
| > \"Instruction Set Architecture for Pervasive Security\".            |
|                                                                       |
| -   Dr. Yuanyuan Yang\'s research result on multicast switching       |
|     networks has been recognized as the currently best-known design   |
|     for non-blocking multicast switching networks in academia and     |
|     industry. Professor Yuanyuan Yang\'s research supported by NSF    |
|     has lead to five patents in the area of interconnection networks, |
|     with three more patents pending.                                  |
|                                                                       |
|     Dr. Yang's research is supported by NSF grants CCR-0073085,       |
|     "Efficient Multicast Communication in Switch-Based Networks for   |
|     Quality of Service", and CCR-0207999, "WDM Optical Interconnect   |
|     Architectures for Parallel and Distributed Computing and          |
|     Communications."                                                  |
|                                                                       |
| -   Dr. Doug Burger and Dr. Stephen W. Keckler have developed a new   |
|     microprocessor architecture which will help performance of        |
|     computer systems to track the trends of Moore\'s law.  This       |
|     grid processor architecture has shown promising performance       |
|     results on desktop, server, and media applications, and is being  |
|     prototyped in partnership with IBM and the United States          |
|     Department of Defense (DARPA).                                    |
+-----------------------------------------------------------------------+
| **B.3 OUTCOME GOAL for TOOLS: Providing "broadly accessible,          |
| state-of-the-art and shared research and education tools."**          |
|                                                                       |
| **Comments:** The program has also been successful in this regard,    |
| and we have identified the following relevant information:            |
|                                                                       |
| -   The Liberty Research Group at Princeton University has developed  |
|     the Liberty Simulation Environment (LSE). LSE is a simulator      |
|     builder designed to reduce barriers in the research of radical    |
|     hardware systems by reducing the cost and improving the fidelity  |
|     of simulation models. LSE is already in use in academic and       |
|     industrial labs including UC Berkeley, Rice, UIUC, Princeton,     |
|     Intel, and Infineon. More information can be found at             |
|     <http://liberty.princeton.edu/>.                                  |
|                                                                       |
|     Dr. August is recipient of NSF CAREER award CCR 0133712,          |
|     "Systematic Design Space Exploration"                             |
|                                                                       |
| -   Dr. Todd Austin and his group at the University of Michigan have  |
|     developed the MASE microarchitectural modeling simulator for      |
|     dynamic verification modeling and reliability analysis. The       |
|     software is available along with the well-known SimpleScalar tool |
|     set previously developed by the investigator.                     |
|                                                                       |
|     Dr. Austin is the recipient of NSF Career award, CCR-0093044,     |
|     "New Directions in Speculative Execution".                        |
|                                                                       |
| -   Professor Narayanan's research has resulted in the development of |
|     the SimplePower Energy Simulator Tool. The simulator has been     |
|     made publicly available and downloaded by 280 research groups     |
|     from both industry and academia across 21 countries.\             |
|     \                                                                 |
|     Professor Narayanan is the recipient of NSF CAREER award          |
|     CCR-0093085, "Energy Efficient Architectures and Their            |
|     Interaction with Software: A Java Perspective"                    |
|                                                                       |
| -   Dr. Padma Raghavan has developed DSCPACK, a package for solving   |
|     large sparse linear systems on advanced architectures. The        |
|     software is in active use at DOE laboratories and NASA.           |
|                                                                       |
|     Dr. Raghavan is a co-PI on NSF grant CCR- 0075792, "Towards and   |
|     Automated Design Environment for Parallel Computing with          |
|     Reconfigurable Processing Elements".                              |
+-----------------------------------------------------------------------+

##### 

##### 

##### **C. OTHER TOPICS**

**C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.**

**C.2 Please provide comments as appropriate on the program's
performance in meeting program-specific goals and objectives that are
not covered by the above questions.**

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

**C.4 Please provide comments on any other issues the COV feels are
relevant.**

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

The subcommittee feels that Nanosystems and Nanocomputers are of
particular importance for the Computer Systems Architecture area, given
the imminence of the technology limitations which will force us to look
at devices beyond silicon. The subcommittee therefore particularly
appreciates the collaboration between CSA and other programs such as
Design Automation for Micro and Nanosystems.

Breakthroughs often occur at the border between established areas, and
large inter-disciplinary projects are known to yield significant
results. Also, prototyping is sometimes essential in systems research
projects, to demonstrate an innovative, revolutionary concept and to
face and overcome the many barriers to its practical implementation.
This means that multidisciplinary projects, prototyping projects, and
ambitious risk-taking ideas should be encouraged, even if such projects
require more funding. However, this should not come at the expense of
other mainstream proposals which form the bloodline of steady research
advances in the CSA area. Hence, appropriate additional resources should
be allocated to encourage risk-taking, large and ambitious projects.

The uneven trends in improvement between processor speed and memory
latency will dramatically exacerbate the design issues of memory
systems. The memory wall problem will essentially dominate the design of
future architectures. Therefore, the subcommittee feels that more
emphasis is needed to address the memory wall problem.

In the same vein, processor design should not eclipse the need for fast
network and storage units. This aspect of research appears to have waned
in recent years at CSA. These issues could be handled in collaboration
with other NSF units.

Finally, benchmarking and performance evaluation have been all too often
ignored by researchers. There is a dire need for a "holistic" approach
to the problem by designing performance metrics and benchmarks which
will emphasize end-to-end performance in future Internet
transaction-based workloads using highly shared hardware infrastructure.
This goal may be contradictory to the usual goal of improving single
program performance that the CSA research community has often focused
on. The future Internet transaction-based workloads will spur new
research on virtualization, dynamic adaptation and optimization, dynamic
self-healing, dynamic power conservation, and quality of service in a
highly shared hardware infrastructure environment. Such topics are all
very relevant to the CSA area, and should be encouraged.

Last but not least, the subcommittee would like to reiterate that budget
is clearly an issue: since only one third of "competitive" proposals are
eventually funded, many worthy projects are obviously ignored by NSF.

**SIGNATURE BLOCK:**

\_ELECTRONICALLY SUBMITTED\_

For the CSA VOV

Jean-Luc Gaudiot

Chair

# Operating Systems and Compilers (OSC)

##### **FY 2003 REPORT TEMPLATE FOR**

#####  **NSF COMMITTEES OF VISITORS (COVs)**

  -----------------------------------------------------------------------
  **Date of COV: June 5-6, 2003**

  **Program/Cluster: DSR**

  **Division: CCR**

  **Directorate: CISE**

  **Number of actions reviewed by COV**[^24]**:\
  Awards: Declinations: Other:**

  **Total number of actions within Program/Cluster/Division during period
  being reviewed by COV**[^25]**: Awards: Declinations: Other:**

  **Manner in which reviewed actions were selected:\
  **
  -----------------------------------------------------------------------

**A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

**A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures..**

+----------------------------------------------------------+-----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES** | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Is the review mechanism appropriate? (panels, ad hoc     |           |
| reviews, site visits)                                    |           |
|                                                          |           |
| Comments: The panel-based review mechanism is            |           |
| appropriate and fair, and shifts the burden of           |           |
| hard-decision making from the PD to a peer-review        |           |
| committee. This is particularly effective in allowing    |           |
| reviewers to be able to reevaluate their reviews during  |           |
| the panel discussion, and breaking ties among proposals  |           |
| that are ranked to be in the same equivalence class. The |           |
| PD is still able to exert his judgment in award          |           |
| decisions by taking into consideration other intangibles |           |
| such as gender, race, and geography and nature of the    |           |
| institutions that a particular PI comes from.            |           |
+----------------------------------------------------------+-----------+
| Is the review process efficient and effective?           |           |
|                                                          |           |
| Comments: For the most part the review process is        |           |
| effective and efficient, and the reviews are usually     |           |
| consistent with the NSF prescribed criteria for proposal |           |
| evaluation. The subcommittee felt that a template        |           |
| describing what goes into each part of a review would    |           |
| help maintain consistency across reviews.                |           |
+----------------------------------------------------------+-----------+
| Are reviews consistent with priorities and criteria      |           |
| stated in the program's solicitations, announcements,    |           |
| and guidelines?                                          |           |
|                                                          |           |
| Comments: While the comments from the reviewers are      |           |
| often elaborate, the subcommittee felt that often there  |           |
| was not a clear correlation between the rating of a      |           |
| proposal and the comments. Further, the reviews are      |           |
| often not broken down into the different categories      |           |
| clearly (such as intellectual merit, broader impact,     |           |
| etc.). This is particularly irksome if a proposal is     |           |
| rejected.                                                |           |
+----------------------------------------------------------+-----------+
| Do the individual reviews (either mail or panel) provide |           |
| sufficient information for the principal investigator(s) |           |
| to understand the basis for the reviewer's               |           |
| recommendation?                                          |           |
|                                                          |           |
| Comments: The subcommittee felt that the panel summaries |           |
| should be more clearly directed towards helping the PI   |           |
| improve his/her proposal especially in case of           |           |
| rejection.                                               |           |
+----------------------------------------------------------+-----------+
| Do the panel summaries provide sufficient information    |           |
| for the principal investigator(s) to understand the      |           |
| basis for the panel recommendation?\                     |           |
| Comments: The fast-lane submission process, the panel    |           |
| format of decision-making, and the interactive panels    |           |
| help streamline the process resulting in a timely        |           |
| feedback to the PIs.                                     |           |
+----------------------------------------------------------+-----------+
| Is the documentation for recommendations complete, and   |           |
| does the program officer provide sufficient information  |           |
| and justification for her/his recommendation?            |           |
|                                                          |           |
| Comments:                                                |           |
+----------------------------------------------------------+-----------+
| Is the time to decision appropriate?                     |           |
|                                                          |           |
| Comments:                                                |           |
+----------------------------------------------------------+-----------+
| Discuss issues identified by the COV concerning the      |           |
| quality and effectiveness of the program's use of merit  |           |
| review procedures:                                       |           |
|                                                          |           |
| Overall the subcommittee felt comfortable with the       |           |
| panel-based merit review process. We present some        |           |
| general comments towards improving the process even      |           |
| more.                                                    |           |
|                                                          |           |
| There are some inherent weaknesses in the panel system.  |           |
|                                                          |           |
| 1\) Insufficient participation of industrial researchers |           |
| on panels.                                               |           |
|                                                          |           |
| 2\) Insufficient participation of west coast researchers |           |
| on panels (at least the panels we looked at).            |           |
|                                                          |           |
| 3\) The set of panelists assembled may not always have   |           |
| the expertise needed to evaluate a given proposal.       |           |
|                                                          |           |
| 4\) The conflict of interest policy of NSF could be      |           |
| overly restrictive especially in highly competitive      |           |
| programs such as the ITR.                                |           |
|                                                          |           |
| 5\) Large programs such as DSC tend to have multiple     |           |
| panels. For example in FY01 there were three panels (2   |           |
| OS panels, and 1 compiler panel). The two OS panels had  |           |
| 13 and 14 proposals each, while the compiler panel had 7 |           |
| proposals. The panel deliberations were independent and  |           |
| from the documentation provided there is no indication   |           |
| that there was a merge of the three panel deliberations  |           |
| to get a global order of fundable proposals.             |           |
+----------------------------------------------------------+-----------+

Here are some concrete ideas to address these weaknesses:

1\) Currently, the panel is **always** held in Washington DC. We
postulate that this could be a primary reason for the insufficient
participation of west coast and industry researchers. We propose moving
the venue of the panels to different locations to enable such
participation. Another possibility is having such panels dovetail some
major conferences (such as SOSP, OSDI, and PLDI) at the location of
these conferences.

2\) Panel membership is often dictated by scheduling constraints, lack
of conflict of interest, etc., resulting in panels whose membership may
lack sufficient expertise for some proposals under consideration.
Therefore, in addition to the panel reviews, the PD could solicit one or
two reviews from domain experts if there is insufficient expertise among
the panelists assembled. Such *ad hoc* reviews should be solicited
**prior** to the panel itself so that such input can be incorporated
into the panel discussion. For example, one proposal was rated
**excellent** by 3 reviewers and **fair** by one reviewer. The excellent
reviews did not have sufficient details to warrant the ratings of the
individual reviews. The panel summary ranked the proposal
**competitive**, and the proposal was funded. As another example,
another proposal had only 2 reviews (**good**, and **very good**). The
panel ranked the proposal **competitive** and the proposal was declined.

3\) We don't see any reason to decide *a priori* how many proposals to
fund in each area (compilers vs. distributed systems/OS). If in a
particular year there are more meritorious proposals in one area than
another, the PD should be willing to shift funding from one area to
another.

4\) Popular initiatives such as ITR should allow construction of panels
that have enough seniority and expertise in evaluating the merits of
large and medium proposals. For example, the subcommittee felt that the
current COI policy precludes most top researchers in the field from
serving on the medium-scale ITR panels since they may have co-authored a
proposal. A solution is to use the services of such researchers on
programs are the ideal vehicles for NSF to launch innovative and
multidisciplinary initiatives with potentially huge impact on the
research output. Naturally such initiatives will draw the most talented
researchers to compete for the awards as well. Therefore it is important
to develop a reasonable merit review process that does not preclude the
use of the services of this talent pool for the evaluation process.

5\) When there is a need to have multiple panels, overlapping the panels
would allow a merge at the end of the individual deliberations.

panels different from the one(s) to which they may have applied. For
example, if a PI has an ITR pending in DSC he/she could be on the panel
for networking or architecture. We bring this point up since it is the
feeling of this subcommittee that such focused

**A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers.**

+-----------------------------------------------------------+----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**           | **YES,   |
|                                                           | NO,**    |
|                                                           |          |
|                                                           | **DATA   |
|                                                           | NOT      |
|                                                           | AV       |
|                                                           | AILABLE, |
|                                                           | or NOT   |
|                                                           | APPL     |
|                                                           | ICABLE** |
+-----------------------------------------------------------+----------+
| Have the individual reviews (either mail or panel)        |          |
| addressed whether the proposal contributes to both merit  |          |
| review criteria?                                          |          |
|                                                           |          |
| Comments: The individual reviews did not always address   |          |
| both the merit review criteria.                           |          |
|                                                           |          |
| Many reviewers don't seem to have a clear idea on what to |          |
| put down for broader impact. For example, one reviewer    |          |
| says, "proposed work will lead to the education of at     |          |
| least 2 PhD students and will help create or develop      |          |
| graduate courses on several fundamental and timely        |          |
| topics", under broader impact.                            |          |
+-----------------------------------------------------------+----------+
| Have the panel summary reviews addressed whether the      |          |
| proposal contributes to both merit review criteria?       |          |
|                                                           |          |
| Comments: The panel summaries tended to cover both merit  |          |
| review criteria much more than the individual reviews.    |          |
| However, there were instances where the panel summary     |          |
| does not document anything pertaining to the broader      |          |
| impact. For example, the panel summary for one proposal   |          |
| (which was funded) says nothing about the broader impact  |          |
| of this proposed research.                                |          |
+-----------------------------------------------------------+----------+
| Have the review analyses (Form 7s) addressed whether the  |          |
| proposal contributes to both merit review criteria?       |          |
|                                                           |          |
| Comments: Leafing through the jackets provided we did not |          |
| see many review analyses forms explicitly mentioning that |          |
| the broader impact criterion was used in the decision     |          |
| making process.                                           |          |
+-----------------------------------------------------------+----------+
| Discuss any issues or concerns the COV has identified     |          |
| with respect to NSF's merit review system.                |          |
|                                                           |          |
| It is clear that many reviewers are not quite sure about  |          |
| the kinds of comments that are appropriate to be          |          |
| documented for the two merit review criteria. It will     |          |
| help a great deal if the review guidelines for reviewers  |          |
| included samples of good and bad examples of comments to  |          |
| put down under broader impact.                            |          |
+-----------------------------------------------------------+----------+

**A.3 Questions concerning the selection of reviewers.**

+---------------------------------------------------------+------------+
| **Selection of Reviewers**                              | **YES,     |
|                                                         | NO,**      |
|                                                         |            |
|                                                         | **DATA NOT |
|                                                         | AVAILABLE, |
|                                                         | or NOT     |
|                                                         | AP         |
|                                                         | PLICABLE** |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       | Yes        |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            | Yes        |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    | Yes        |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      | Yes        |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
|                                                         |            |
| The answers to most of the questions in this section    |            |
| are in the affirmative. However, it is the feeling of   |            |
| this subcommittee that determination of **COI** should  |            |
| be segregated by areas so that experts can serve on     |            |
| important panels such as ITR even if they have          |            |
| submitted proposals for consideration in a given        |            |
| program.                                                |            |
+---------------------------------------------------------+------------+

**A.4 Questions concerning the resulting portfolio of awards under
review.**

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     |               |
| projects supported by the program.                   |               |
|                                                      |               |
| Comments: Career awards were of a very high quality; |               |
| however, regular awards encourage mediocrity since   |               |
| limited budget limits the scope of the proposed      |               |
| research. Interesting systems projects (such as      |               |
| Clouds at Georgia Tech, Eden at U of Washington,     |               |
| Psyche at U of Rochester, Charlotte at U of          |               |
| Wisconsin) of yesteryears have involved multiple PIs |               |
| and a large number of graduate and undergraduate     |               |
| students. All these projects have been funded by NSF |               |
| and have had a tremendous impact on the              |               |
| state-of-the-art of building operating systems and   |               |
| distributed systems. The current base-level funding  |               |
| mechanism in the DSC program with average size of    |               |
| \$300K per award just does not encourage significant |               |
| research in distributed systems. Consequently it is  |               |
| hard to find too many projects funded by the base    |               |
| program that are high-risk, or multidisciplinary.    |               |
| Many of the career proposals tended to be            |               |
| innovative. The regular base level proposals are     |               |
| mostly incremental in nature due to the limited      |               |
| funding to expect a lot of innovation. On the other  |               |
| hand, several ITR proposals were quite innovative    |               |
| and multidisciplinary.                               |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  |               |
| scope of the projects?                               |               |
|                                                      |               |
| Comments: The subcommittee felt that on an average   |               |
| the size of the awards is inadequate for systems     |               |
| research. The duration of the award is fine.         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| High Risk Proposals?                                 |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of: Multidisciplinary Proposals?             |               |
|                                                      |               |
| Comments: I                                          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Innovative Proposals?                                |               |
|                                                      |               |
| Comments: The projects funded through this program   |               |
| address critical needs as cited in reports such as   |               |
| PITAC. Issues addressed by the funded projects       |               |
| include sensor based distributed systems, high       |               |
| performance cluster and grid systems, and reliable   |               |
| distributed computing.                               |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Funding for centers, groups and awards to            |               |
| individuals?                                         |               |
|                                                      |               |
| Comments: This program does not fund centers. The    |               |
| program has allocated roughly a third of the base    |               |
| budget to career awards and thus has a good number   |               |
| of new investigators funded in the program. There    |               |
| appears to be a good geographical distribution of    |               |
| the awards as well as across institution types       |               |
| through RUI and SGER programs. The number of funded  |               |
| projects to underrepresented minorities is not as    |               |
| high as it could be but it is the belief of this     |               |
| subcommittee that this is not unique to this program |               |
| but perhaps a more systemic problem that needs to be |               |
| addressed at the level of the agency as a whole.     |               |
| Please refer to Part C under "other topics".         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Awards to new investigators?                         |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Geographical distribution of Principal               |               |
| Investigators?                                       |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Institutional types?                                 |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Projects that integrate research and education?      |               |
|                                                      |               |
| Comments: It is hard to assess if there is a         |               |
| significant number of projects funded by the program |               |
| that integrate research and education. However, many |               |
| funded proposals state that they will infuse         |               |
| research results into the curriculum.                |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance:                                             |               |
|                                                      |               |
| Across disciplines and subdisciplines of the         |               |
| activity and of emerging opportunities?              |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          |               |
| participation of underrepresented groups?            |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      |               |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
|                                                      |               |
| Current budget for the program is totally            |               |
| insufficient. The program needs a balance between    |               |
| small and large projects. If not for the infusion of |               |
| ITR money in this area, this program will be         |               |
| ineffective. For example, the base budget dropped    |               |
| from \$7.8M to \$6.3M from FY01 to FY02. With an     |               |
| out-year commitment of \$3.6 M, the PD effectively   |               |
| had only \$2.7M for making new awards in this        |               |
| program for FY02! The ITR infused \$8.6M in FY02     |               |
| breathing new life into the program.                 |               |
+------------------------------------------------------+---------------+

**A.5 Management of the program under review**

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments: The program has been managed very well with a shoestring    |
| budget. The PD has had to make hard decisions with a huge influx of   |
| new requests for funding.                                             |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments:                                                             |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments:                                                             |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| The subcommittee makes the following fairly obvious suggestions to    |
| improve the management of the program:                                |
|                                                                       |
| 1\) Increase staff support, so that the PD is able to devote some     |
| time to think about ways in which he/she can make the program more    |
| adaptive to novel research directions.                                |
|                                                                       |
| 2\) Make more rational allocation of funds to this program taking     |
| into account the experimental and high resource needs of this program |
| relative to other programs in CCR.                                    |
+-----------------------------------------------------------------------+

##### B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.**

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

+-----------------------------------------------------------------------+
| **[B.1 NSF OUTCOME GOAL for PEOPLE:]{.underline}** **Developing "a    |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| Below we present examples of some of the people-development outcomes  |
| of this program.                                                      |
|                                                                       |
| Prof. Mor Harchol-Balter, under her Career Award 0133077, is studying |
| scheduling of requests in overloaded web servers. She has             |
| demonstrated that scheduling server requests using SRPT               |
| (shortest-remaining-processing-time first) can significantly improve  |
| performance and reduce average response time by up to a factor of 5.  |
|                                                                       |
| In another Career Award (0133456), Prof. Andrea Arpaci-Dusseau of     |
| University of Wisconsin is attacking the problem of simplifying the   |
| extension of large and complex operating systems using gray-box       |
| services. The idea is to provide new services through a layer that    |
| exists between client applications and the operating                  |
|                                                                       |
| system. Without modifying OS code, the gray-box services use          |
| knowledge about OS algorithms and behavior to provide, e.g., new      |
| information to applications.                                          |
|                                                                       |
| Professor s Lorenzo Alvisi, an assistant professor at the University  |
| of Texas, Austin (Career CCR 9734185), and Norman Ramsey, an          |
| assistant professor at Harvard University (Career CCR 9733974) , were |
| named Alfred P. Sloan Research Fellow for 2001-2003. Selection        |
| procedures for the fellowships are designed to show "the most         |
| outstanding promise of making fundamental contributions to new        |
| knowledge." 16 past Sloan Fellows have become Nobel Laureates; 13     |
| have won the Fields Medal in Mathematics. The fellowship carries an   |
| award of \$40,000 for the period. Alvisi's research focuses on        |
| developing recovery-based techniques that combine high performance    |
| during failure-free executions with fast recovery. The extensive      |
| literature in this area seldom uses rollback recovery to build        |
| reliable distributed applications. The Lightweight Fault-Tolerance    |
| project changes this state of affairs with an approach that blends    |
| algorithm work, system building, and empirical analysis. The results  |
| will have a significant impact on distributed agent-based computing.  |
| Ramsey's research is to develop a framework for reusable              |
| specifications of machine instructions' semantics. This research      |
| addresses the problem of retargeting machine-level tools used to      |
| translate, analyze, instrument, improve, and migrate programs. It can |
| also be extended to the development of tools of both scientific and   |
| practical importance. The Sloan Research Fellowships were based on    |
| the recognition of their research, which are being supported by NSF   |
| CAREER Awards.                                                        |
|                                                                       |
| Three students in the University of Wisconsin WiND research project,  |
| headed by Professors Andrea and Remzi Arpaci-Dusseau (Career CCR      |
| 0133456 and CCR 0092840). broke a world record in external sorting    |
| using a cluster of PCs. The Datamation sort has long been an          |
| industrial standard database benchmark, with the challenge being to   |
| sort one million 100-byte records as quickly as possible. The         |
| previous record was 0.998 seconds on a single machine with special    |
| hardware built for fast sorting. In contrast, the students' results   |
| came on a platform of 32 commodity PCs running the Linux operating    |
| system, all connected by an Intel Gigabit Ethernet switch. The new    |
| record, as set by Ina Popovici, John Bent, and Brian Forney, is 0.48  |
| seconds, halving the previous mark. This research was funded by a     |
| number of NSF grants, including a Research Infrastructure award,      |
| which put the hardware in place, and a CCR grant for research and     |
| WiND project storage systems.                                         |
+-----------------------------------------------------------------------+
| **[B.2 NSF OUTCOME GOAL for IDEAS:]{.underline} Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| As an example of a grant that may have major impact, Career Award     |
| 0133302 to Prof. Andrew Meyers of Cornell University seeks to use     |
| language- and compiler-based mechanisms to improve confidentiality of |
| information in computer networks. The approach uses type systems and  |
| static analysis of code. Verifying that programs protect              |
| confidentiality is clearly a crucial problem in the modern internet.  |
| Several tools have been made available for download, including the    |
| Jiff language/compiler framework.                                     |
|                                                                       |
| Another significant problem today is software piracy. Professor       |
| Christian Colberg (grant 0307249) has been studying software          |
| watermarking and tamper-proofing techniques that can serve as         |
| protection against piracy and reverse engineering.                    |
|                                                                       |
| Professor Klara Nahrstedt,University of Illinois, Urbana-Champaign,   |
| (CCR 9988199, 9972884) applies the results from the integration of a  |
| research project on QoS mapping (CCR 9988199) and facilities          |
| supported by an infrastructure grant (CCR 9972884) to distant         |
| learning applications. The objective of the research project is       |
| design a framework for mapping QoS requirements at different system   |
| and application levels. Her research team has developed a QoS         |
| programming environment with XML-like language that allow users to    |
| annotate multimedia components with their applications and system QoS |
| parameters such as frame rate, frame size, resolution, bandwidth,     |
| delay, loss and others. This idea has been received as a basis for    |
| the Service Level Agreement (SLA) by the Open Group standard          |
| committee. The impact of this research has been demonstrated in a     |
| distant learning application at UIUC where video lectures are         |
| delivered to various locations with Smart Room devices to provide     |
| augmented realities in an ubiquitous computing environment. In this   |
| application the differentiated routing is deployed such that premium  |
| traffic goes through a different path than best effort traffic.       |
|                                                                       |
| Professor Seth Goldstein, Carnegie Mellon University, (CAREER CCR     |
| 9876248) is investigating compiler and architecture designs for very  |
| large reconfigurable fabrics constructed from molecular electronics.  |
| This research examines issues in designing nano-scale systems at many |
| levels including new devices, new circuits, fabrication friendly      |
| architectures, and compilation. It is an important step toward        |
| building nano-scale systems. The work is a spin-off from an NSF       |
| CAREER grant in the C-CR Division.                                    |
|                                                                       |
| The research in reconfigurable computing based on electronic          |
| nanotechnology has had an impact on physical scientists (chemists,    |
| physicists, material scientists, etc.) as well as computer scientists |
| and electrical engineers. Goldstein's work has led to computing       |
| device designs that are more amenable to chemical assembly and more   |
| scalable fabrication. In particular, a two-terminal replacement for   |
| the transistor is expected to lead to more easily manufactured        |
| molecular electronics. Designs for large-scale reconfigurable fabrics |
| should have an impact on sub-100nm electronics as well as             |
| nanotechnology. This research opens up a great opportunity for        |
| collaborative research between computer science and physical science  |
| and engineering.                                                      |
+-----------------------------------------------------------------------+
| Principle investigator Michael Scott, Unversity of Rochester, (CCR    |
| 0204344) and his research teams at the University of Rochester have   |
| developed a middleware, InterWeave, that allows programs running on   |
| machines anywhere on the Internet to                                  |
|                                                                       |
| share memory as if they were running on the same machine. Sharable    |
| memory is organize into segments that are managed by InterWeave       |
| servers, much as web pages are managed by HTTP servers. This system   |
| dramatically simplifies the construction of                           |
|                                                                       |
| distributed applications. The figure shows how InterWeave augments    |
| the Astroflow stellar dynamics simulator, which is currently deployed |
| in interactive kiosks at science museums around the country, to allow |
| real-time visualization of simulations at remote                      |
|                                                                       |
| sites. To accomplish this type of distributed computation in a        |
| network environment, InterWeave addresses research issues such as     |
| system interoperability and data coherency. The system now runs on    |
| many platforms including Pentium, Alpha, MIPS,                        |
|                                                                       |
| Sparc, and Power-PC-based computers, with several variants of         |
| Unix/Linux and Windows.                                               |
|                                                                       |
| Principle investigators Ling Liu and Calton Pu, Georgia Institute of  |
| Technology, CCR 9988452) introduced the concept of Continual Querie   |
| and have developed several techniques to support efficient processing |
| of continual queries that monitor interesting                         |
|                                                                       |
| events using distributed triggers and notify the user of changes      |
| whenever updates of interest happen. The continual query project has  |
| produced two operational systems: **Open CQ** for monitoring          |
| semi-structured information updates, and **Web CQ** for monitoring    |
| changes in arbitrary Web pages. The Web CQ system has been suggested  |
| and used by National Cancer Institute to track cancer clinical trial  |
| information over a dozen information sources. Web CQ helps cancer     |
| researchers, patients, friends, and                                   |
|                                                                       |
| relatives track new treatments and new cancer trials of interest.     |
|                                                                       |
| The research develops algorithms on differential reevaluation of      |
| continual queries by using an incremental approach to monitoring      |
| information updates. This research has led to the current wide        |
| interest in data stream research. The CQ technologies is being        |
| applied to a number of application areas including logistics and      |
| unified access to about 500 biological databases. Calton Pu became a  |
| Fellow of AAAS (American Association for Advancement of Sciences) for |
| his contributions in system software and transaction management       |
| research, which includes work supported by NSF/CCR.                   |
|                                                                       |
| **B.3 OUTCOME GOAL for TOOLS: Providing "broadly accessible,          |
| state-of-the-art and shared research and education tools."**          |
|                                                                       |
| Comments: Principle investigator Michael Scott, Unversity of          |
| Rochester, (CCR 0204344) and his research teams at the University of  |
| Rochester have developed a middleware, InterWeave, that allows        |
| programs running on machines anywhere on the Internet to share memory |
| as if they were running on the same machine. Sharable memory is       |
| organize into segments that are managed by InterWeave servers, much   |
| as web pages are managed by HTTP servers. This system dramatically    |
| simplifies the construction of distributed applications. The figure   |
| shows how InterWeave augments the Astroflow stellar dynamics          |
| simulator, which is currently deployed in interactive kiosks at       |
| science museums around the country, to allow real-time visualization  |
| of simulations at remote sites. To accomplish this type of            |
| distributed computation in a network environment, InterWeave          |
| addresses research issues such as system interoperability and data    |
| coherency.                                                            |
+-----------------------------------------------------------------------+

The system now runs on many platforms including Pentium, Alpha, MIPS,
Sparc, and Power-PC-based computers, with several variants of Unix/Linux
and Windows.

Principle investigators Ling Liu and Calton Pu, Georgia Institute of
Technology (CCR 9988452) introduced the concept of Continual Querie and
have developed several techniques to support efficient processing of
continual queries that monitor interesting events using distributed
triggers and notify the user of changes whenever updates of interest
happen. The continual query project has produced two operational
systems: **Open CQ** for monitoring semi-structured information updates,
and **Web CQ** for monitoring changes in arbitrary Web pages. The Web CQ
system has been suggested and used by National Cancer Institute to track
cancer clinical trial information over a dozen information sources. Web
CQ helps cancer researchers, patients, friends, and relatives track new
treatments and new cancer trials of interest. The research develops
algorithms on differential reevaluation of continual queries by using an
incremental approach to monitoring information updates. This research
has led to the current wide interest in data stream research. The CQ
technologies is being applied to a number of application areas including
logistics and unified access to about 500 biological databases. Calton
Pu became a Fellow of AAAS (American Association for Advancement of
Sciences) for his contributions in system software and transaction
management research, which includes work supported by NSF/CCR.

##### 

##### 

#####  **C. OTHER TOPICS**

**C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.**

The program does an excellent job of funding high-quality young research
ers through the Career grant program. We believe that this expenditure,
currently about 1/3 of the program budget, is a worthwhile investment in
bootstrapping the research careers of the best new PhDs in the area.

**C.2 Please provide comments as appropriate on the program's
performance in meeting program-specific goals and objectives that are
not covered by the above questions.**

CCR needs to accept the fact that the base programs are, to a large
extent, broken. **The average grant size for many of these programs has
not changed for** **two decades**, while the personnel and overhead
costs have risen significantly. The result is that many base program
grants are unable to fund even the most minimal research effort (e.g.,
summer support for a single PI and one or two graduate students). On the
one hand, an effort is being made to maintain support for as many
researchers as possible given the constraints, but nobody can maintain a
research effort with such insufficient support. Furthermore, the
programs discourage collaboration, since grant size is often fixed,
independent of the number of PIs.

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

Specifically, research in distributed systems that will have significant
impact requires significant resources, both people (typically multiple
PIs and multiple graduate students) and equipment (e.g., a 32-node
cluster would be a minimal requirement for many potential efforts in
this area). Such research is essentially **forbidden** in the context of
the current program budget. The result is that only small research in
distributed systems can be supported under the base program.

**C.4 Please provide comments on any other issues the COV feels are
relevant.**

For the reasons cited above, ITR has been the saving grace for medium-
to large-scale research. It is crucial that ITR be preserved in some
form in the future. Either ITR should be retained as is, or the ITR
budget should be moved to the base programs. However, in the later case,
the structure of large/medium/small grants **must** be preserved to
ensure that projects of substantial magnitude can be supported. It would
be a huge mistake to turn ITR money into a large number of
lowest-possible-amount grants.

We do not believe that NSFs goals for diversity are being seriously met.

The reasons are complex. To begin with, the pool of minority and
under-represented applicants is exceedingly small. Of the small set of
minorities who are capable of applying for grants, many do not do so. In
many cases, research and grant writing are not encouraged or supported
by the institutions at which many under-represented groups teach (e.g.,
HBCUs Historically Black Colleges and Universities). Therefore, NSF must
be proactive in encouraging under-represented faculty to apply, at
mentoring them through the process, and at ensuring that their
institutions provide sufficient support for research activities (e.g.,
release time, matching funds, etc.).

. There seems to be inequity in the budget allocation within CISE and
within CCR. First of all, among the five divisions within CISE (CCR,
ANI, IIS, ACI, and EIA), CCR fields roughly 40% of all proposals
submitted to CISE. However the funding level for the divisions have been
roughly the same in FY00, 01, and 02. Essentially this means that CCR
has to fund 40% of the proposals submitted to CISE with 20% of the total
CISE budget.

Further, all programs within CCR do not warrant the same level of
funding. Programs such as DSC, SEL, and EHS fund proposals with high
experimental components. Such proposals need a higher level of funding
to have a significant impact compared to proposals that are fielded by
other programs such as TOC, SPS, and DA. Providing funding levels
proportional to the number of proposals in each program within CCR is
NOT the right approach given the qualitative difference for doing
effective research in these different programs.

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the Operating Systems and Compilers \[OSC\]

\[Kishore Ramachandran\]

Chair

# Software Engineering and Languages (SEL)

##### **FY 2003 REPORT TEMPLATE FOR**

#####  **NSF COMMITTEES OF VISITORS (COVs)**

  -----------------------------------------------------------------------
  **Date of COV: June 5-6, 2003**

  **Program/Cluster: SEL**

  **Division: CCR**

  **Directorate: CISE**

  **Number of actions reviewed by COV**[^26]**:\
  Awards: 23 Declinations: 27 Other: 0**

  **Total number of actions within Program/Cluster/Division during period
  being reviewed by COV**[^27]**: 326 Awards: 287 Declinations: 239
  Other:0**

  **Manner in which reviewed actions were selected:\
  **
  -----------------------------------------------------------------------

**A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

The Committee believes that panels are an appropriate review mechanism
for the typical class of research program supported by SEL program. It
appears to be both efficient and effective, although it can be difficult
to obtain the right reviewers for large multidisciplinary projects. The
Committee examined a range of jackets, and evaluated the thoroughness of
the reviews, the thoroughness of the Program Directors' evaluations of
the reviews, and the appropriateness of the Program Directors' decisions
(given the limitation on the large number of proposals submitted and
limited funds available in the SEL). Our opinion is that well-reasoned
decisions have been made, and good feedback has been given to the PIs.
The FASTLANE system works well. It should reduce the response time from
submission to decision.

**A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures..**

+----------------------------------------------------------+-----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES** | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Is the review mechanism appropriate? (panels, ad hoc     |           |
| reviews, site visits)                                    |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Is the review process efficient and effective?           |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Are reviews consistent with priorities and criteria      |           |
| stated in the program's solicitations, announcements,    |           |
| and guidelines?                                          |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Do the individual reviews (either mail or panel) provide |           |
| sufficient information for the principal investigator(s) |           |
| to understand the basis for the reviewer's               |           |
| recommendation?                                          |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Do the panel summaries provide sufficient information    |           |
| for the principal investigator(s) to understand the      |           |
| basis for the panel recommendation?\                     |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Is the documentation for recommendations complete, and   |           |
| does the program officer provide sufficient information  |           |
| and justification for her/his recommendation?            |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Is the time to decision appropriate?                     |           |
|                                                          |           |
| Comments: See Comments Below                             |           |
+----------------------------------------------------------+-----------+
| Discuss issues identified by the COV concerning the      |           |
| quality and effectiveness of the program's use of merit  |           |
| review procedures:                                       |           |
|                                                          |           |
| In the sampling we examined, individual reviews provided |           |
| sufficient information for the principal investigator(s) |           |
| to understand the basis for the reviewer's               |           |
| recommendation.                                          |           |
|                                                          |           |
| Different Program directors have different styles, but   |           |
| the summaries are very useful, providing the principal   |           |
| investigator(s) a good understanding of the strengths    |           |
| and weaknesses of their proposals.                       |           |
|                                                          |           |
| The Program Directors have begun providing a full set of |           |
| panel results for all proposals reviewed during a        |           |
| particular panel. This provides a good view of the       |           |
| relative merit of each proposal within each panel. It    |           |
| facilitates subsequent review of the decisions.          |           |
|                                                          |           |
| The committee believes that decisions should be made     |           |
| within six months of proposal submission deadline. A     |           |
| random selection of jackets showed that this was often   |           |
| but not always the case.                                 |           |
+----------------------------------------------------------+-----------+

**A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers.**

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       |           |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments: The individual reviews addressed the           |           |
| contribution of the proposal to both merit review        |           |
| criteria. The question of intellectual merit and broader |           |
| impact is asked explicitly and the reviewers answer it.  |           |
| For the years that we reviewed, the summary reviews did  |           |
| not routinely address both issues, although the          |           |
| individual reviews almost always did. It was more likely |           |
| to see a discussion of the intellectual merit.           |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments:                                                |           |
+----------------------------------------------------------+-----------+
| Have the review analyses (Form 7s) addressed whether the |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: The Form 7 review analyses are more likely to  |           |
| have some discussion of the broad impact as well as the  |           |
| intellectual merit.                                      |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system.               |           |
|                                                          |           |
| In summary, we believe there will be more discussion of  |           |
| the broad impact in the future since this is now being   |           |
| emphasized. The Committee thinks this is a good idea.    |           |
| FASTLANE should explicitly facilitate that both issues   |           |
| be addressed on all appropriate forms.                   |           |
+----------------------------------------------------------+-----------+

**A.3 Questions concerning the selection of reviewers.**

+---------------------------------------------------------+------------+
| **Selection of Reviewers**                              | **YES,     |
|                                                         | NO,**      |
|                                                         |            |
|                                                         | **DATA NOT |
|                                                         | AVAILABLE, |
|                                                         | or NOT     |
|                                                         | AP         |
|                                                         | PLICABLE** |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       |            |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments:                                               |            |
|                                                         |            |
| The SEL Program Directors have consistently asked for   |            |
| at least 4 reviews per proposal. When adequate reviews  |            |
| have not been achieved via the panel they have asked    |            |
| for external reviews. The reviewers had appropriate     |            |
| expertise and/or qualifications. There appears to be a  |            |
| separate reviewer mix for the PL and SE aspects of the  |            |
| program -- more specifically separate panels -- which   |            |
| is important as the two areas requires much different   |            |
| types of expertise. In fact, Software Engineering is    |            |
| such a wide-ranging topic as it matures, that it may    |            |
| require split panels dealing with different aspects of  |            |
| the discipline.                                         |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            |            |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments: The SEL Program Directors been very proactive |            |
| in soliciting a good mix of panelists: a mix of junior, |            |
| mid-career and veteran researchers, including           |            |
| minorities and women. They recognize and resolve        |            |
| conflicts of interest when appropriate. The minutes of  |            |
| the meetings show that conflict of interest issues were |            |
| taken seriously and followed closely.                   |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    |            |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      |            |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
|                                                         |            |
| Some panels had no industry representation. It is       |            |
| especially useful on SE panels to have participants who |            |
| have both research and development experience.          |            |
+---------------------------------------------------------+------------+

**A.4 Questions concerning the resulting portfolio of awards under
review.**

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     |               |
| projects supported by the program.                   |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  |               |
| scope of the projects?                               |               |
|                                                      |               |
| Comments: The awards are appropriate in size and     |               |
| duration for the scope of the programming language   |               |
| projects, but not for most software engineering      |               |
| proposals (see below). Most proposals appear to be   |               |
| short term, incremental improvements on existing SEL |               |
| research. It is assumed that the high risk proposals |               |
| go to the ITR program.                               |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| High Risk Proposals?                                 |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of: Multidisciplinary Proposals?             |               |
|                                                      |               |
| Comments: The program and funding discourage         |               |
| submission of multidisciplinary proposals. The       |               |
| program specifically funds focused projects on       |               |
| software engineering and programming languages.      |               |
| Multidisciplinary proposals even within software     |               |
| engineering must go to the ITR program as their      |               |
| scope is beyond that of the SEL program and thus     |               |
| they fall between the cracks of the existing NSF     |               |
| program structure.                                   |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Innovative Proposals?                                |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Funding for centers, groups and awards to            |               |
| individuals?                                         |               |
|                                                      |               |
| Comments: The committee believes that the program    |               |
| advances research in appropriate areas. But, the     |               |
| level of funding does not allow for funding of       |               |
| centers and groups. That also has been the ITR       |               |
| purview.                                             |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Awards to new investigators?                         |               |
|                                                      |               |
| Comments: There is an appropriate balance of awards  |               |
| to new investigators, mostly through the CAREER      |               |
| award program. However, the committee believes that  |               |
| the mandatory minimum of CAREER awards should vary   |               |
| by discipline to take maximum advantage of available |               |
| funding and allow for the support of more young      |               |
| scientists.                                          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Geographical distribution of Principal               |               |
| Investigators?                                       |               |
|                                                      |               |
| Comments: There was no evidence of a problem in      |               |
| geographic distribution. Most proposals come from    |               |
| Ph.D. granting institutions. Some grants went to     |               |
| EPSCoR states and minority institutions. The percent |               |
| of women and minorities PIs is above the national    |               |
| averages in Computer Sciences.                       |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Institutional types?                                 |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Projects that integrate research and education?      |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance:                                             |               |
|                                                      |               |
| Across disciplines and subdisciplines of the         |               |
| activity and of emerging opportunities?              |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          |               |
| participation of underrepresented groups?            |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      |               |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments: Clearly, the program is relevant to        |               |
| national priorities. Software is pervasive to our    |               |
| modern society. Some areas, such as the              |               |
| *foundations* of software engineering and            |               |
| programming languages generally do not have other    |               |
| sources of funding to pursue.                        |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
+------------------------------------------------------+---------------+

**A.5 Management of the program under review**

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments: We believe the program has been well managed over the past  |
| three years, as can be seen by our earlier comments. The Program      |
| Directors through their position in the community, attendance at      |
| conferences and workshops, are able to stay current with emerging     |
| research trends. In addition, by selecting panelists, who are also    |
| active researchers and educators, the program stays responsive to     |
| emerging trends in research and education.                            |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments: As an example of the program responding to emerging         |
| research trends, the SEL sponsored a workshop on New Visions for      |
| Software Design & Productivity: Research & Applications at Vanderbilt |
| University to develop a long-term agenda for software engineering     |
| research. Of particular interest were technology needs and promising  |
| solutions that can revolutionize the way we design and produce        |
| software in the coming decades, but that are currently beyond the     |
| scope of today's time-to-market and profit-driven research and        |
| development (R&D) programs. The workshop addressed the following four |
| issues central to software design and productivity: The Future of     |
| Software and Software Research, New Software Development Paradigms,   |
| Software for the Real World, and Software for Large-scale             |
| Network-Centric Systems.                                              |
|                                                                       |
| As an example of the program responding to emerging education trends, |
| John Hale, an NSF CAREER award recipient who directs the Tulsa        |
| University's Center of Excellence in Cybersecurity, established a     |
| certificate program at TU on information security. The program        |
| involves protecting Internet systems from hackers and                 |
| cyberterrorists. TU has extended its national leadership role in      |
| battling cybercrime by becoming the only American university that     |
| offers five government-endorsed certificates in information security, |
| an emerging national priority.                                        |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments:                                                             |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| We believe it is important that the staff is funded to stay current   |
| with emerging research trends, including the ability to travel to     |
| relevant conferences and workshops, as well as the time to plan and   |
| prioritize important research topics in the area.                     |
+-----------------------------------------------------------------------+

##### B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were mad**e.
**Incremental progress made on results reported in prior fiscal years
may also be considered.**

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

B.  **Please provide comments on the activity as it relates to NSF's
    Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
    appropriate. Examples should reference the NSF award number, the
    Principal Investigator(s) names, and their institutions.**

**[B.1 NSF OUTCOME GOAL for PEOPLE:]{.underline}** **Developing "a
diverse, internationally competitive and globally engaged workforce of
scientists, engineers, and well-prepared citizens."**

Comments:

SEL PIs are at some of the best CS departments in the country (CMU,
UT-Austin, UC-Berkeley, MIT, etc.), at far-flung institutions in EPSCoR
states (U Idaho, Kansas St, U SE Maine, U Wyoming -- Laramie, etc), at
small universities and colleges (Loyola of Baltimore, Loyola of Chicago,
Bates College, Drexel U, U Tulsa, etc.), and at minority institutions
(Florida International U, NM St, etc.) In fact, over the past 4 years,
SEL has received proposals from 42 states and funded projects in 30
states.

a)  SEL PIs include Native Americans (John Hale at U Tulsa), Black
    Americans (Rod Moten at Colgate U and France at Colorado St), and
    Hispanics (Jose Meseguer at SRI), as well as foreign-born Americans
    from many countries (India, China, Czech Republic, UK, So Africa,
    etc). Nonetheless, the numbers of PIs from underrepresented groups
    are small, fitting the profile of these groups in computer science
    research more broadly.

b)  Of the 160 awards still active in June of this year, 24, or 15%, are
    to female PIs.

c)  Of the 38 awards made in FY2000, 9, or 24%, are to female PIs.

d)  Most of the awards given by SEL support one or more graduate
    students

e)  Over the 3 years 7 SEL awards and 4 ITR awards in the SEL area went
    to EPSCOR states in 2000

**SEL supports the development of a diverse workforce only through
research grants, which in turn support graduate students and some
curricular development and modification.**

+-----------------------------------------------------------------------+
|                                                                       |
+-----------------------------------------------------------------------+
| **[B.2 NSF OUTCOME GOAL for IDEAS:]{.underline} Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| **Comments:**                                                         |
|                                                                       |
| 1.  George Necula, UC-Berkeley, CCR-9875171, designed the CCured      |
|     compiler for C that improves the execution speed of programs by   |
|     exploiting the fact that most C programs are type safe.. Since C  |
|     programs are in common use and the language does not guarantee    |
|     type safety this has an impact on real software development.      |
|                                                                       |
| 2.  Allen Emerson of UT-Austin (CCR- 9804736) has developed a         |
|     provably efficient algorithm for rendering the familiar and       |
|     classical timing diagram notation into the framework of temporal  |
|     logic. This allows temporal logic expressions, unlike the         |
|     original diagrams, to be fed directly into verification tools     |
|     that check the diagrams for consistency and other properties      |
|     given by the designer, thus avoiding tedious and error prone      |
|     translation or doing verification by hand.                        |
|                                                                       |
| 3.  Philip Johnson of U Hawaii -- Manoa (CCR- 9804010), showed that   |
|     programmer "guesstimates" for project planning were consistently  |
|     better than any of the automatically derived estimates using a    |
|     variety of analytical tools for automatic generation of project   |
|     planning information. This suggests that rather than perform      |
|     research for a \"holy grail\" estimation procedure, it might be   |
|     more practical and effective to provide a suite of simpler        |
|     estimation tools to help developers make \"informed\" guesses.    |
|                                                                       |
| 4.  Nancy Leveson of MIT, CCR-9633428, developed the concept of       |
|     "Intent Specifications" for software systems and has rewritten    |
|     the official FAA specification of TCAS II as an intent            |
|     specification. Intent specifications integrate formal and         |
|     informal methods into a system requirements specification         |
|     methodology based on fundamental research in human problem        |
|     solving and employing a new type of hierarchical abstraction      |
|     based on intent or design rationale.                              |
+-----------------------------------------------------------------------+
| **B.3 OUTCOME GOAL for TOOLS: Providing "broadly accessible,          |
| state-of-the-art and shared research and education tools."**          |
|                                                                       |
| Comments: While not specifically funding infrastructure, many SEL     |
| projects produce languages, programming environments, or tools for    |
| testing, verification, measurement, or other software engineering     |
| activities. These are invariably made available to the research       |
| community via the web and/or other distribution methods. Outstanding  |
| examples include Lambda-Prolog, Proof-Carrying Code technology, Typed |
| Assembly Language (TAL), and visualization software. More detailed    |
| examples include:                                                     |
|                                                                       |
| a)  Catalin Roman, Washington U, CCR-9970939, designed Lime           |
|     middleware to facilitate rapid development of mobile applications |
|     over wired or wireless networks in settings that involve physical |
|     mobility of hosts, logical mobility of code, or a combination of  |
|     the two. Lime was released for public use under an open source    |
|     license this past year. Through a collaboration with Ford Motor   |
|     Co. this technology is being transferred to the automotive        |
|     industry with the goal of developing highway applications such as |
|     collision warning systems, information sharing among vehicles,    |
|     epidemic software upgrades (e.g., pollution reduction protocols), |
|     safety analysis across the entire fleet, highway traffic          |
|     optimization, etc.                                                |
|                                                                       |
| b)  Susan Horwitz, U Wisconsin, CCR-9987435 used program slicing as a |
|     basis for the design and implementation of a tool that finds      |
|     duplicated code and extracts it into a procedure. The tool has    |
|     been used successfully to identify duplicate code in a number of  |
|     Unix utilities, as well as a graph-layout program used in-house   |
|     by IBM.                                                           |
|                                                                       |
| c)  Edmund Clarke of CMU, CCR-9803774, has developed a Symbolic Model |
|     Checker. It remains the most successful method yet devised for    |
|     formally verifying that hardware or software systems meet their   |
|     specifications. It has successfully uncovered any number of       |
|     subtle errors in systems such as cache consistency controllers,   |
|     CPU circuits, and networking protocols, and has been adapted by   |
|     such companies as Intel, Motorola, IBM, and Siemens, for hardware |
|     verification.                                                     |
|                                                                       |
| d)  Terry Swift, SUNY-Stony Brook, CAREER award (CCR- 9702681),       |
|     developed a table logic program, called XSB, that has opened new  |
|     applications areas for logic programming in data cleaning and     |
|     integration, psychiatric diagnosis, ontology management, web      |
|     agents, verification of concurrent systems, circuit diagnosis,    |
|     machine learning. To date three companies use XSB as core         |
|     technology: Knowledgebus, Inc (Hanover, Md) MD, Logix (Towson,    |
|     Md), and XSB, Inc (Stony Brook, NY).                              |
|                                                                       |
| e)  Gary Leavens, Iowa St. U., CCR-9803843, is developing a new       |
|     open-source behavioral interface specification language for Java, |
|     JML, that allows vendors to ship object-oriented libraries and    |
|     frameworks without the source code that is usually needed due to  |
|     the difficulty of documenting and reasoning about inheritance.    |
|     JML 4.0 has recently been released for widespread use and is      |
|     available through SourceForge.                                    |
|                                                                       |
| f)  Annie Antón; North Carolina State University; CCR-9983926; has    |
|     developed a set of tools that help ensure that privacy policies   |
|     are aligned with the software systems that they govern. One such  |
|     tool to support the discovery, elaboration and management of      |
|     system use scenarios for the validation of software requirements  |
|     and policies, is SMaRT (Scenario Management and Requirements      |
|     Tool).                                                            |
+-----------------------------------------------------------------------+

##### **C. OTHER TOPICS**

**C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.**

One issue for SEL is the mix of two areas of research, Software
Engineering and Programming Languages, which require different panels,
have different approaches to research (e.g., most SE research requires
empirical study, some form of tool use or development, and deals with
larger size problems), and have different criteria for evaluation. The
SE community tends to be more diverse in its research opportunities and
therefore points of view of its panelists. For example, it was not
uncommon to see particular SE proposals rated with grades varying from E
to F, e.g., conflicting ratings. There was more consistency among the PL
ratings.

The committee feels that, given the nature of software engineering
research, the basic traditional grant funding and mechanism needs to be
reconsidered. First, much of the work in areas like software engineering
(and systems) requires empirical work and other forms of validation;
implementation is often involved. Second, software engineering research
requires collaboration among several researchers in various other
software engineering sub-disciplines to solve the big problems, as well
as interdisciplinary collaborations, if it is going to have the broad
impact and long range effect required by society (e.g., network-centric
secure systems of systems, advanced, large-scale applications). It
requires a staffing profile that includes post-docs, junior and senior
researchers in collaboration. The small size and limited staffing for
the typical SE award makes it less likely that PIs will be able to make
significant new contributions to the field. One or two graduate student
type grants will not have the effect needed. The cost of this type of
program cannot be taken out of the existing SEL program and the current
programming language research support. It requires funding mechanisms of
the kind that have been supported by the ITR program.

The committee is concerned that the lack of the appropriate type of
software engineering funding mechanisms, along with the current low
percent of funded proposals in software engineering (this rate seems to
be lower than other groups) will have negative impact on the discipline.
Researchers will be rewarded for doing the wrong kind of research and
fewer young Ph.D.s will go into the discipline.

Note: The idea that large corporate research organizations will do the
basic research required for the evolution of the discipline is
misguided. Basic research funding is critical to advancing the field as
a whole and is not necessarily in Company X's best interest.

On a relative scale, the size of CAREER grants is large compared to the
size of grants typically received by senior researchers. Young
researchers can use these grants to work on small pieces of the large
problem, but what kind of research can they do when they evolve to
working on the bigger classes of problems?

**C.2 Please provide comments as appropriate on the program's
performance in meeting program-specific goals and objectives that are
not covered by the above questions.**

The Committee believes the SEL program's performance is limited by the
pairing of programming languages and software engineering into one
program. This affects the selection of panels, the evaluation of very
diverse proposals, the comparison of panel results, and even the
selection of the Program Director.

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

Given the nature of the NSF structure, many big-payoff research areas
require the interaction of several PDs. This creates a problem for a
principal investigator -- where to go with a new proposal, a problem for
creating panels -- how do you create the proper mix and motivate them to
see the other sides of the issues. It is hard to motivate a short term
interdisciplinary activity without imposing a fair amount of interaction
and work on the appropriate PDs. But it appears difficult to add another
time-consuming activity to the already over-burdened PDs. (This
interaction is especially relevant for an enabling discipline like
software engineering given the pervasive nature of software to every
scientific discipline.)

**C.4 Please provide comments on any other issues the COV feels are
relevant.**

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

The COV review could have been improved by giving the committee more
lead time, by more effective briefings by NSF staff on programs, by
better guidance for COV on format and content of report.

Notes:

Data for industry and government being under represented on panels

Criteria for broad impact

It would be nice to capture the debate at the reviews in the summary

The award rate is going down in SEL over the past few years

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the \[SEL COV\]

Victor R. Basili, Chair

Debra Richardson

Anthony Wasserman

# Design Automation for Micro and Nano Systems (DA)

##### **FY 2003 REPORT TEMPLATE FOR**

#####  **NSF COMMITTEES OF VISITORS (COVs)**

  -----------------------------------------------------------------------
  **Date of COV: June 5-6, 2003**

  **Program/Cluster: Design Automation for Micro and Nano Systems**

  **Division: CCR**

  **Directorate: CISE**

  **Number of actions reviewed by COV**[^28]**:\
  Awards: 15 Declinations: 9 Other: 1**

  **Total number of actions within Program/Cluster/Division during period
  being reviewed by COV**[^29]**: 92 Awards: 168 Declinations: 10
  Other:**

  **Manner in which reviewed actions were selected: 28 actions were
  reviewed. 21 jackets were selected by the program for review. Seven
  additional jackets were requested at random by the COV to cover award,
  decline and withdrawal actions.**
  -----------------------------------------------------------------------

**A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

VLSI design methodologies are exceptionally challenged by the rapid
advances in deep submicron, mechanical (MEMS), optical, nano, and
quantum computing media. This program supports basic research underlying
the science and methodologies for designing integrated systems comprised
of micro systems in traditional silicon VLSI technology, in MEMS
technologies, and in computing and communications media of the future.
Since new computing technologies are not expected to replace silicon in
the foreseeable future, research must continue on silicon VLSI design as
well as design in new technologies. The program primarily addresses
three areas: exploration of fundamental questions of how to design in
future computing and communications media; meeting challenges of design
in VLSI silicon as geometries shrink; and investigating design methods
for technologies such as optical, MEMS and analog/mixed signal.

The Design Automation for Micro and Nano Systems has several main focus
areas:

-   **Physical design:** Routing & layout, power optimization, logic
    synthesis, on chip communication, modeling & device simulation

-   **System level design:** Systems on chips, embedded systems,
    application specific processor design, hardware software co-design

-   **Test and verification:** testing of analog, digital mixed signal
    systems, built in self test (BIST), design for testability. formal
    proof of correctness

-   **Multi-technologies:** MEMS, Nanotechnologies, Analog/Mixed Signal
    designs

The COV members noticed and were concerned that there was variability in
the budget for the Design Automation for Micro and Nano Systems Program.
The financial strength of the DA program is critical for the advancement
of design methods for circuits and systems in new and emerging
technologies of national importance. The DA program is strong and
vibrant and is supporting new researchers through CAREER awards and is
planning future directions in nanotechnology through workshops and
recent awards.

**A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures..**

+----------------------------------------------------------+-----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES** | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Is the review mechanism appropriate? (panels, ad hoc     |           |
| reviews, site visits)                                    |           |
|                                                          |           |
| Comments: The DA program uses mostly panel review for    |           |
| proposals with some additional ad hoc mail review. This  |           |
| seems appropriate.                                       |           |
+----------------------------------------------------------+-----------+
| Is the review process efficient and effective?           |           |
|                                                          |           |
| Comments: Yes. A number of specialized panels based on   |           |
| topic area are used to evaluate and rank the proposals.  |           |
+----------------------------------------------------------+-----------+
| Are reviews consistent with priorities and criteria      |           |
| stated in the program's solicitations, announcements,    |           |
| and guidelines?                                          |           |
|                                                          |           |
| Comments: Yes. The reviews were consistent with the      |           |
| programs goals. However, it is felt that the message     |           |
| that circuit design is also encouraged has not been      |           |
| conveyed to prospective PI\'s.                           |           |
+----------------------------------------------------------+-----------+
| Do the individual reviews (either mail or panel) provide |           |
| sufficient information for the principal investigator(s) |           |
| to understand the basis for the reviewer's               |           |
| recommendation?                                          |           |
|                                                          |           |
| Comments: Yes. The reviews do appear to be detailed.     |           |
+----------------------------------------------------------+-----------+
| Do the panel summaries provide sufficient information    |           |
| for the principal investigator(s) to understand the      |           |
| basis for the panel recommendation?\                     |           |
| Comments: Yes. The panel summaries are effective and     |           |
| have improved from FY00 to FY02 to give more information |           |
| on broader impact.                                       |           |
+----------------------------------------------------------+-----------+
| Is the documentation for recommendations complete, and   |           |
| does the program officer provide sufficient information  |           |
| and justification for her/his recommendation?            |           |
|                                                          |           |
| Comments: The documentation was sufficient for the COV   |           |
| to understand the reasons for the recommendations. In    |           |
| the 28 jackets that the COV looked at, awards and        |           |
| declines, the review analysis clearly explained the      |           |
| decision.                                                |           |
+----------------------------------------------------------+-----------+
| Is the time to decision appropriate?                     |           |
|                                                          |           |
| Comments: The time to decision was reasonable for the    |           |
| program considering deadlines and funding availability.  |           |
| For FY00, the time was 7.75 months, for FY01, the time   |           |
| was 6.05 months, and for FY02 the time was 4.37months.   |           |
| This is a good trend and it is felt that NSF Fastlane    |           |
| has helped here.                                         |           |
+----------------------------------------------------------+-----------+
| Discuss issues identified by the COV concerning the      |           |
| quality and effectiveness of the program's use of merit  |           |
| review procedures:                                       |           |
|                                                          |           |
| The PD exercised good judgment in incorporating          |           |
| innovative proposals into the program. The PD analyzed   |           |
| the reviews in all cases and also funded some high risk  |           |
| proposals in newer areas. This is to be encouraged.      |           |
+----------------------------------------------------------+-----------+

**A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers.**

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       |           |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments: We have seen steady improvement in the reviews |           |
| from FY00 to FY02 in addressing both review criteria. In |           |
| general, the first criterion on intellectual merit is    |           |
| better addressed than the second criteria.               |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: Also, in the panel summary, there has been an  |           |
| improving trend in addressing both criteria. Again, the  |           |
| broader impact criterion has not always been addressed.  |           |
+----------------------------------------------------------+-----------+
| Have the review analyses (Form 7s) addressed whether the |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: It is the understanding of the COV that        |           |
| starting in FY03, the review analysis forms should all   |           |
| consider both merit criteria. In looking at the          |           |
| FY00-FY02 jackets, the second was not always addressed.  |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system.               |           |
|                                                          |           |
| It appears the reviewers have not always fully           |           |
| understood the meaning and purpose of the broader impact |           |
| criterion.                                               |           |
+----------------------------------------------------------+-----------+

**A.3 Questions concerning the selection of reviewers.**

+---------------------------------------------------------+------------+
| **Selection of Reviewers**                              | **YES,     |
|                                                         | NO,**      |
|                                                         |            |
|                                                         | **DATA NOT |
|                                                         | AVAILABLE, |
|                                                         | or NOT     |
|                                                         | AP         |
|                                                         | PLICABLE** |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       |            |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments: Yes. The program has a large group of reviews |            |
| with at least three reviewers per proposal in the 28    |            |
| jackets that were reviewed by the COV.                  |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            |            |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments: The program had a good choice of reviewers    |            |
| with academic and where appropriate industrial          |            |
| reviewers. In the more established area of VLSI CAD,    |            |
| the reviewer names are mostly familiar to people in the |            |
| field. In the emerging cross-disciplinary areas of      |            |
| nanotechnology and in the NSF/SRC initiative on         |            |
| mixed-signal devices some names were not familiar to    |            |
| the members of the COV but this may be due to the broad |            |
| nature of the proposals.                                |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    |            |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments: The review base was well balanced based on    |            |
| geography and institution groups.                       |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      |            |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments: Yes. The Program Director has addressed       |            |
| procedures for identifying COI in the review analysis   |            |
| and in the panel minutes.                               |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
+---------------------------------------------------------+------------+

**A.4 Questions concerning the resulting portfolio of awards under
review.**

  ----------- ------------ -------------- --------------- ------------------- ------------- -----------
  **Year**    **Awards**   **Declines**   **Withdrawn**   **Inappropriate**   **Total       **Success
                                                                              Proposals**   Rate**

  **FY00**    29           57             3               0                   89            32.58%

  **FY01**    44           49             6               0                   99            44.44%

  **FY02**    19           62             0               1                   82            23.17%

  **Total**   92           168            9               1                   270           34.07%
  ----------- ------------ -------------- --------------- ------------------- ------------- -----------

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     |               |
| projects supported by the program.                   |               |
|                                                      |               |
| Comments: From looking at the 28 jackets selected    |               |
| and looking at the awards and declines, the COV was  |               |
| impressed by the overall quality of the proposals    |               |
| selected for award in areas of VLSI                  |               |
| computer-aided-design, mixed-signal design, and      |               |
| nanotechnology.                                      |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  |               |
| scope of the projects?                               |               |
|                                                      |               |
| Comments: The award size was reasonable given the    |               |
| constraints. Compared to ITR funded projects, the    |               |
| regular awards were small. Overall, there was        |               |
| reduction in funds in the base program that covers a |               |
| critical area in electrical and computer             |               |
| engineering.                                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| High Risk Proposals?                                 |               |
|                                                      |               |
| Comments: There were a small number of high risk     |               |
| proposals but efforts were being made to encourage   |               |
| new areas.                                           |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of: Multidisciplinary Proposals?             |               |
|                                                      |               |
| Comments: The NSF/SRC mixed-signal initiative and    |               |
| the NERS nanotechnology initiative were excellent    |               |
| programs with industry and other programs in NSF. .  |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Innovative Proposals?                                |               |
|                                                      |               |
| Comments: This seems appropriate.                    |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Funding for centers, groups and awards to            |               |
| individuals?                                         |               |
|                                                      |               |
| Comments: There were insufficient funds for centers  |               |
| but two medium ITR's were awarded. The rest of the   |               |
| program funded single investigator and small group   |               |
| awards.                                              |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Awards to new investigators?                         |               |
|                                                      |               |
| Comments: The program puts a priority on CAREER      |               |
| awards having funded 18 proposals of 37 submitted    |               |
| over the FY00-FY02 period. There were fewer awarded  |               |
| in FY02 due to the budget limitation in the DA       |               |
| program.                                             |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Geographical distribution of Principal               |               |
| Investigators?                                       |               |
|                                                      |               |
| Comments: There are several cooperative              |               |
| international awards to projects with Mexico,        |               |
| Brazil, China, and Taiwan. There is an appropriate   |               |
| peak in the number of awards to California,          |               |
| partially due to the large number of design          |               |
| activities in that state. Also, there was an award   |               |
| with EPSCOR funding.                                 |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Institutional types?                                 |               |
|                                                      |               |
| Comments: No RUI proposals were received by the DA   |               |
| program. However, there were a number of awards to   |               |
| smaller research universities.                       |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Projects that integrate research and education?      |               |
|                                                      |               |
| Comments: The program was generous in awarding REU   |               |
| supplements for undergraduate research. Besides      |               |
| regular DA proposals, there are CAREER, INT,         |               |
| CONACyT, CNPq, and other special program awards.     |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance:                                             |               |
|                                                      |               |
| Across disciplines and subdisciplines of the         |               |
| activity and of emerging opportunities?              |               |
|                                                      |               |
| Comments: Yes. The program funds activities          |               |
| including nanotechnology, MEMS, mixed-signal, and    |               |
| VLSI CAD. It is felt that more emphasis could be     |               |
| placed on circuit design (not necessarily automated) |               |
| for analog/mixed-signal and emerging technologies.   |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          |               |
| participation of underrepresented groups?            |               |
|                                                      |               |
| Comments: There is progress in the number of females |               |
| receiving awards in Design Automation. Three out of  |               |
| 18 CAREER awards were made to women, which is very   |               |
| encouraging. All of them are making strong           |               |
| contributions to the field.                          |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      |               |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments: The Design Automation program is critical  |               |
| to national needs for advanced integrated systems.   |               |
| The SIA Roadmap grand challenges and workforce needs |               |
| all point to areas of priority in the Design         |               |
| Automation program. The PI's funded by DA are making |               |
| major contributions that are reflected by best paper |               |
| awards and patents, through IEEE and ACM Fellow      |               |
| distinctions, and in textbooks and articles in       |               |
| EE-Times.                                            |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
|                                                      |               |
| Overall, the program has been excellent in spite of  |               |
| the fact that the PD changed often until a permanent |               |
| Program Director was selected in FY03.               |               |
+------------------------------------------------------+---------------+

**A.5 Management of the program under review**

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments: Since the retirement of Dr. Robert Grafton, the management  |
| of the program has been somewhat lacking due to the absence of a      |
| Program Director. There was a sequence of temporary program directors |
| over the last two years. The situation is corrected with the new PD,  |
| Dr. Sankar Basu, who began in October 2002.                           |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments: We feel that the program is eager to respond to new trends; |
| that is evidenced by the NSF/SRC initiative in mixed signal systems,  |
| and the initiatives in MEMS and nano technology. There is less of an  |
| opportunity in educational trends; but the individual PI's have been  |
| successful in educational activities through textbooks and community  |
| interaction and programs for women. They were not directly funded for |
| these activities; however, many of the PI's are active in new         |
| educational techniques.                                               |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments: Overall, the planning activities were somewhat bumpy due to |
| the transition to a new program director. The recent workshop on      |
| Nanotechnology (Oct. 2002) shows planning for new activities in the   |
| program.                                                              |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| There was an uneven amount of funds provided to the program during    |
| the three years. This was especially true with ITR funding in FY2001. |
| This was probably due to the lack of a Program Director and the       |
| creation of two new programs in CCR, but it is a major concern. Funds |
| should stabilize and grow for this important program.                 |
+-----------------------------------------------------------------------+

##### B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.**

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

+-----------------------------------------------------------------------+
| **[B.1 NSF OUTCOME GOAL for PEOPLE:]{.underline}** **Developing "a    |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| Comments: Design Automation researchers are recognized leaders in     |
| their fields. Seven (7) PI's funded by the Design Automation program  |
| were elected IEEE Fellows, and one (1) PI was elected ACM Fellow      |
| during the period FY2000-FY2002. Also, Design Automation PI's were    |
| awarded a Technical excellence award from SRC, an Alfred P. Sloan     |
| Research Fellowship, and a Naval Research Young Investigator Award.   |
| In particular:                                                        |
|                                                                       |
| CCR/DA Project Title: CAD Techniques for Embedded Video Signal        |
| Processing (CCR-9912413)                                              |
|                                                                       |
| Institution: Princeton University                                     |
|                                                                       |
| PI/coPI Name: Wayne Wolf                                              |
|                                                                       |
| Wayne Wolf was elected a Fellow of the ACM in 2002. Wayne also        |
| received the Terman Award by the American Association for Engineering |
| Education (ASEE) for this textbook on VLSI Design.                    |
|                                                                       |
| **CCR/DA Project Title: Low-Energy Datapath Design for Programmable   |
| Digital Signal Processors (CCR-9988262)**                             |
|                                                                       |
| Institution: University of Minnesota, Twin Cities                     |
|                                                                       |
| PI/coPI Name: Keshab Parhi                                            |
|                                                                       |
| **Keshab Parhi has been awarded the IEEE Kiyo Tomiyasu Technical      |
| Field Award for "pioneering contributions to high-speed and low-power |
| digital signal processing architectures for broadband communication   |
| systems."**                                                           |
|                                                                       |
| CCR/DA Project Title: Fast 3-D Analysis and Macromodel Generation of  |
| Interconnect, Packaging and MEMS Using Green\'s Function Independent  |
| Accelerated Iterative Methods (CCR-9901195)                           |
|                                                                       |
| Institution: MIT                                                      |
|                                                                       |
| PI/coPI Name: Jacob White                                             |
|                                                                       |
| One of Jacob White's students, Rebecca-Jing Li has been awarded the   |
| Householder Award from the Applied Math community and society, SIAM,  |
| for the best Ph.D. dissertation in numerical linear algebra and its   |
| applications to model reduction of large MEMS systems.                |
|                                                                       |
| Also, the following international grants were active and monitored by |
| the Design Automation Program and overlapped with the period under    |
| review. The CONACyT program is for collaboration with researchers in  |
| Mexico, whereas the CNPq program is for collaborations with           |
| researchers in Brazil                                                 |
|                                                                       |
|   --------- -                                                         |
| -------- --------- ----------- -------------------------- ----------- |
|   CONACyT                                                             |
|   9711099   Ramirez   NM State    Built In Test Techniques   \$99,380 |
|                                                                       |
|                                           for Mixed-Mode VLSI         |
|                                                                       |
|   CNPq                                                                |
|  9901628   Devadas   MIT         NSF CNPq Collaborative     \$231,823 |
|                                                                       |
|                                           Research on Design          |
|                                                                       |
|                                           Environments for            |
|                                                                       |
|                                           Application-Specific        |
|                                                                       |
|                                           Programmable Processors     |
|                                                                       |
|   CNPq                                                                |
|  9900881   Malik     Princeton   NSF-CNPq Collaborative     \$225,000 |
|                                                                       |
|                                           Research on Design          |
|                                                                       |
|                                           Environments for            |
|                                                                       |
|                                           Application-Specific        |
|                                                                       |
|                                           Programmable Processors     |
|   --------- -                                                         |
| -------- --------- ----------- -------------------------- ----------- |
+-----------------------------------------------------------------------+
| **[B.2 NSF OUTCOME GOAL for IDEAS:]{.underline} Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| **Comments:** There were nine (9) best paper awards in journals (4)   |
| and conferences (5), as well as two patents during this period by     |
| researchers supported by the Design Automation program. The program   |
| has fostered an interest in the problem of designing in the emerging  |
| nanotechnology medium. A number of nanotechnology NER grants          |
| transferred to the DA program in this frontier area. The program is   |
| promoting design in MEMS and related technologies (e.g. optical)      |
| through a variety of mechanisms: SGERs, regular grants, and a         |
| workshop that was held in the fall of 2000.                           |
|                                                                       |
| **CCR/DA Project Title: CAREER: Robust Behavioral Fault Simulation    |
| Algorithms for Multilevel Simulation (CCR-9733584)**                  |
|                                                                       |
| Institution: Tufts University                                         |
|                                                                       |
| PI/coPI Name: Karen Panetta                                           |
|                                                                       |
| Prof. Karen Panetta is making contributions to both research and the  |
| environment at Tufts. She arrived at Tufts eight years ago, at the    |
| start of the dean\'s diversity initiative. She had worked for Digital |
| Equipment Corp., the computer maker, where she was used to being the  |
| only woman in the room. When she arrived at Tufts, she was the only   |
| woman on the engineering faculty. Today, there are six women, four of |
| them on a tenure track, out of about two dozen professors. The key to |
| her survival was the dean, who kept asking what she needed to         |
| succeed, she said.                                                    |
|                                                                       |
| ![](media/image17.jpeg){width="2.588888888888889in"                   |
| height="1.3465277777777778in"}                                        |
|                                                                       |
| **CCR/DA Project Title: SGER -- A VLSI-like Design Methodology for    |
| Nanoelectromechanical Systems (NEMS) (CCR-0107623)**                  |
|                                                                       |
| Institution: University of Illinois at Urbana-Champaign               |
|                                                                       |
| PI/coPI Name(s): N. R. Aluru                                          |
|                                                                       |
| Website: <http://www.staff.uiuc.edu/~aluru/>                          |
|                                                                       |
| NSF support for this project led to a fundamental understanding on    |
| the physical and engineering aspects of nanoelectromechanical systems |
| (NEMS). Specifically, in this project, Prof. Aluru has developed      |
| atomistic simulation tools, and combined them with continuum theories |
| to develop multiscale simulation tools for accurate analysis of NEMS. |
| He has identified the length scales at which classical physical       |
| theories break-down for NEMS -- multiscale simulation tools prove     |
| valuable when classical continuum theories are not accurate for       |
| design and design automation of NEMS. Aluru has also taken a big step |
| in creating compact models for operation of NEMS. These compact       |
| models will enable the creation of a VLSI-like design methodology for |
| NEM arrays and circuits.                                              |
+-----------------------------------------------------------------------+
| **CCR/DA Project Title: CAREER: Integrated Computational MEMS (IC     |
| MEMS) (CCR-9875671)**                                                 |
|                                                                       |
| Institution: University of Illinois at Urbana-Champaign               |
|                                                                       |
| PI/coPI Name(s): N. R. Aluru                                          |
|                                                                       |
| Website: <http://www.staff.uiuc.edu/~aluru/>                          |
|                                                                       |
| NSF support for this project led to revolutionary new ideas,          |
| specifically algorithms, for radically simpler and easier analysis of |
| microelectromechanical systems (MEMS). In this project, efficient     |
| computational methods for scattered point and meshless analysis of    |
| electrostatic MEMS are developed. Electrostatic MEM devices are       |
| governed by coupled mechanic al and electrostatic energy domains. A   |
| self-consistent analysis of electrostatic MEMS was developed by       |
| combining a finite cloud method based interior mechanical analysis    |
| with a boundary cloud method based exterior electrostatic analysis.   |
| Lagrangian descriptions are used for both mechanical and              |
| electrostatic analysis. Prof. Aluru has shown that meshless finite    |
| cloud and boundary cloud methods combined with fast algorithms and    |
| Lagrangian descriptions are flexible, efficient and attractive        |
| alternatives compared to conventional finite element/boundary element |
| methods for self-consistent electromechanical analysis. The Illinois  |
| research group has analyzed and designed several electrostatic MEMS   |
| devices using the tools developed in this project. One Ph.D. student, |
| one women M.S. student, and one undergraduate minority student have   |
| been supported on this                                                |
| project.![](media/image18.jpeg){width="2.5854166666666667in"          |
| height="2.4604166666666667in"}                                        |
+-----------------------------------------------------------------------+

**B.3 OUTCOME GOAL for TOOLS: Providing "broadly accessible,
state-of-the-art and shared research and education tools."**

Comments: In the area of tools and accessibility, researchers supported
by the Design Automation program have made many contributions to the
field. This has been primarily through the creation and distribution of
Computer-Aided-Design (CAD) software tools. Since Design Automation is a
very international activity, there has been strong international
collaboration with PRC, Taiwan, Brazil, and Mexico. Also, the
Semiconductor Research Corporation of the Semiconductor Industry
Association has worked closely with the DA program on a successful joint
initiative on Mixed Signal design.

Several patents (Hayes, Michigan and Gupta, Irvine), a startup company
(Shanbhag & Singer, U Illinois), and several textbooks have also been
results of DA supported research in the area of tools. In the area of
educations, many Ph.D. (typically 2/project/yr) research students are
supported, and many undergraduate students are involved in research
(typically 20 REUs/yr). Also, the DA program supports a number of
workshops, such as the Nanocomputing Workshop at CMU, CCR/DA Project
Title: Giga-Scale System-On-A-Chip Design (CCR-0096383)

Institution: University of California, Los Angeles

PI/coPI Name: Jason Cong

Website:
[**http://cadlab.cs.ucla.edu/\~pubbench/placement/**](http://ballade.cs.ucla.edu/~pubbench/placement/)

Supported under the NSF Award CCR-0096383, the researchers at UCLA led
by Prof. Jason Cong have made significant progress on optimality study
of existing placement algorithms for integrated circuits. Using a set of
cleverly constructed circuit examples that match industrial circuit
characteristics and have known optimal solutions, researchers from UCLA
show that the results of leading placement tools from both industry and
academia (including, for example, Qplace from Cadence) are 70% to 150%
(!) away from the optimal solutions in terms of the total wirelength.
This is the first time a quantitative evaluation is provided for the
optimality of existing placement algorithms. Note that if such a large
optimality gap can be closed, it will be equivalent to several
technology generation advancements (in comparison, the introduction of
copper interconnects is equivalent to a 30% wirelength reduction, and so
is each process technology scaling, but each requires *multi-billion
dollar* investment). This research signifies the opportunity and
importance of further research in physical design for large-scale
integrated circuits.

This result was first reported at ASPDAC\'2003 in Japan.  It soon
generated great interest from both the research community and
semiconductor/EDA industry. It was featured as the cover story in the EE
Times, a leading trade journal, on February 5, 2003. The examples used
in this study (named PEKO examples) have been made available on the
Internet at
[http://cadlab.cs.ucla.edu/\~pubbench/placement/](http://ballade.cs.ucla.edu/~pubbench/placement/)
and have been downloaded by researchers and engineers from over 40
companies and universities worldwide within one week since the result
was covered by EE Times.

CCR/DA Project Title: IP-Based Embedded Systems Design (CCR-9876006) and
Collaborative Research: Platform-Based CAD for Power and Performance
Optimization (CCR-0203829)

Institution: UC
Riverside![](media/image19.jpeg){width="2.0236111111111112in"
height="2.69375in"}

PI/coPI Name: Frank Vahid

Website: <http://www.cs.ucr.edu/~vahid>

This project emphasizes new design methods to deal with today's
high-capacity embedded computer chips. The most fundamental change in
required methods is a unified view of hardware and software. The project
develops methods for tuning highly configurable system-on-a-chip
designs, including memory reconfiguration and hardware/software
partitioning , to specific embedded applications.

As a broader impact of this research, there was the publication of an
undergraduate textbook titled \"Embedded System Design: A Unified
Hardware/Software Introduction\" by Frank Vahid and Tony Givargis,
published by J. Wiley and Sons, Oct 2001. This book is presently being
used in numerous universities and its use is spreading. The book is the
first to elevate embedded system design to the level of design
principles, rather than just low-level assembly language programming. It
presents software and digital design in a unified way, showing that they
are fundamentally the same, differing primarily in their design metrics
like performance, power and design effort. Web site is
<http://www.cs.ucr.edu/esd>.

Another output is \"Platune,\" a tool developed by Tony Givargis as part
of his Ph.D. research, which is currently being utilized as the
foundation for research, and for teaching, in several other
universities. Platune is a \"Platform exploration\" tool, which enables
one to quickly tune a system-on-a-chip platform\'s configurable
parameters to best fit any particular application. The tool is Windows
based and is very easy to use. One simply provides the application
program that one want to execute on the platform, and the tool
automatically performs an efficient search through the massive
configuration space, providing the Pareto-optimal set of configurations
in a nice graphical format. Platune was covered in an IEEE Transactions
on CAD article. Other researchers have begun to use the tool as the
basis for their own research and are extending the tool. Some people are
also using it in their advanced courses. Web site is
<http://www.cs.ucr.edu/~dalton/Platune/>.

#####  **C. OTHER TOPICS**

**C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.**

The program is now in the right direction with the new Program Director
who started this year. The PD has a strong vision and direction for this
program and is very receptive to new ideas.

The COV feels that the addition of Micro and Nano Systems to the title
of the program is very appropriate for the new direction of the program.
However, in certain cases for design automation to be possible, design
ideas and techniques must be developed first. Therefore, the COV feels
that design activities (not necessarily design automation) should also
be encouraged.

The COV notes with pleasure the very broad nature of the program,
ranging from devices to systems. However, there appears to be a gap:
Although circuits (which consist of clusters of devices, often connected
in very sophisticated ways) are the building blocks of systems, the
program does not seem to explicitly include circuits work. The COV feels
that circuits work should be properly emphasized, as it forms the bridge
between devices and systems, Many new developments will be needed in
circuits, especially in non-traditional technologies such as deep
submicron, nanotechnology, MEMS, and analog/mixed-signal.

**C.2 Please provide comments as appropriate on the program's
performance in meeting program-specific goals and objectives that are
not covered by the above questions.**

Previously the DA program had great impact through the MOSIS chip
fabrication service for education and research. As we know, MOSIS has
been funded by NSF, DARPA, and most recently by industry, through the
Semiconductor Research Corporation and the Semiconductor Industry
Association. There is a critical national need to continue chip
fabrication for students and researchers in Electrical Engineering and
Computer Science, and possibly this should be brought under the DA
program in the future. To keep competitive, it should be noted that the
European community has access to advanced fabrication facilities and
design kits through Eurochip,
([http://www.eurochip.com](http://www.eurochip.com/)). We should
preserve the MOSIS infrastructure, improve the interface to foundries
through better design kits, improve the interface for MEMS fabrication,
and begin planning for a nanotechnology version of MOSIS. Additional
funds for the Support of MOSIS are a tangible way to enhance the
educational impact of the DA program.

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

NSF should consider using pre-proposal white papers in more of the
programs to give more flexibility to program directors and to reduce the
preparation burden of full proposals on proposers especially junior
investigators.

C.4 Please provide comments on any other issues the COV feels are
relevant.

Feedback of oral panel discussions to the proposers could be better
presented to assist the PI's in preparing a revision. Often key points
may be lost in the panel summary transcription. This could be an
opportunity for new technology.

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

A template for the final form would have been appreciated in addition to
the draft data collection template. A Fastlane-type form on the web
could also be an alternative. Also program material, possibly on CD-ROM,
could be sent in advance to make review more effective.

**SIGNATURE BLOCK:**

Joseph R. Cavallaro

Robert W. Newcomb

Yannis P. Tsividis

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the \[Design Automation for Micro and Nano Systems (DA)\]

\[Joseph R. Cavallaro, Sub-committee Chair of COV\]

Chair

# Communications Research (COMM)

##### **FY 2003 REPORT TEMPLATE FOR**

#####  **NSF COMMITTEES OF VISITORS (COVs)**

  -----------------------------------------------------------------------
  **Date of COV: June 5-6, 2003**

  **Program/Cluster: Communications Research**

  **Division: CCR**

  **Directorate: CISE**

  **Number of actions reviewed by COV**[^30]**:\
  Awards: 19 Declinations: 20 Other:**

  **Total number of actions within Program/Cluster/Division during period
  being reviewed by COV**[^31]**: Awards: Declinations: Other:**

  **Manner in which reviewed actions were selected:\
  **
  -----------------------------------------------------------------------

**A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

**A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures..**

+----------------------------------------------------------+-----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES** | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Is the review mechanism appropriate? (panels, ad hoc     | Yes       |
| reviews, site visits)                                    |           |
|                                                          |           |
| Comments: The combination of panels and ad-hoc reviews   |           |
| give the program director sufficient information to      |           |
| decide the awards. The composition of the panels is      |           |
| judged as qualified.                                     |           |
+----------------------------------------------------------+-----------+
| Is the review process efficient and effective?           | Yes       |
|                                                          |           |
| Comments: The current procedure is effective in          |           |
| identifying the best proposals for funding.              |           |
+----------------------------------------------------------+-----------+
| Are reviews consistent with priorities and criteria      | Yes       |
| stated in the program's solicitations, announcements,    |           |
| and guidelines?                                          |           |
|                                                          |           |
| Comments: The COV recognizes the program provides        |           |
| guidelines to the reviewers regarding priorities and     |           |
| criteria stated in the program documentation and the     |           |
| program documentation and the reviews are consistent     |           |
| with those priorities and criteria.                      |           |
+----------------------------------------------------------+-----------+
| Do the individual reviews (either mail or panel) provide | Yes       |
| sufficient information for the principal investigator(s) |           |
| to understand the basis for the reviewer's               |           |
| recommendation?                                          |           |
|                                                          |           |
| Comments: Although the COV recognizes that it is         |           |
| difficult to impose a strict set of rules to enforce     |           |
| each and every review complies with this rule, the COV   |           |
| believes that this requirement is satisfied by the       |           |
| guidelines provided by the program. The overall reviews  |           |
| provide ample input to the PI.                           |           |
+----------------------------------------------------------+-----------+
| Do the panel summaries provide sufficient information    | Yes       |
| for the principal investigator(s) to understand the      |           |
| basis for the panel recommendation?\                     |           |
| Comments: The COV believes that the panel process        |           |
| results in effective summaries for the PI regarding      |           |
| feedback from ad-hoc reviews and the panel.              |           |
+----------------------------------------------------------+-----------+
| Is the documentation for recommendations complete, and   | Yes       |
| does the program officer provide sufficient information  |           |
| and justification for her/his recommendation?            |           |
|                                                          |           |
| Comments: The COV believes that the current              |           |
| documentation process is thorough.                       |           |
+----------------------------------------------------------+-----------+
| Is the time to decision appropriate?                     | Yes       |
|                                                          |           |
| Comments: The COV did not come across any reason to      |           |
| believe that the time to decision is inappropriately     |           |
| long or short.                                           |           |
+----------------------------------------------------------+-----------+
| Discuss issues identified by the COV concerning the      |           |
| quality and effectiveness of the program's use of merit  |           |
| review procedures:                                       |           |
|                                                          |           |
| The COV considers the program effective and of good      |           |
| quality in the use of its merit review procedures.       |           |
+----------------------------------------------------------+-----------+

**A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers.**

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       | Yes       |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments: The program is successful when reviewers       |           |
| address the element of both generic criteria. The COV    |           |
| believes however that not all PIs can articulate the     |           |
| Broader Impact criterion well. Also, at times, the       |           |
| importance of that criterion is not clear. The COV       |           |
| believes that the panel should be able to make this      |           |
| judgement under the guidance of the Program Director,    |           |
| and it has come to the conclusion that the program       |           |
| currently achieves this task.                            |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     | Yes       |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: The COV observed that the panel summaries      |           |
| occasionally addressed Criterion 2. The COV believes     |           |
| this criterion is not always relevant and considers the  |           |
| level of emphasis by the program appropriate.            |           |
+----------------------------------------------------------+-----------+
| Have the review analyses (Form 7s) addressed whether the | No        |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: The current forms do not address both merit    |           |
| criteria.                                                |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system.               |           |
|                                                          |           |
| The COV considers the merit review of the program        |           |
| sufficiently detailed and appropriately steered.         |           |
+----------------------------------------------------------+-----------+

**A.3 Questions concerning the selection of reviewers.**

+---------------------------------------------------------+------------+
| **Selection of Reviewers**                              | **YES,     |
|                                                         | NO,**      |
|                                                         |            |
|                                                         | **DATA NOT |
|                                                         | AVAILABLE, |
|                                                         | or NOT     |
|                                                         | AP         |
|                                                         | PLICABLE** |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       | Yes        |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments: The COV believes the program made use of a    |            |
| sufficient number of referees.                          |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            | Yes        |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments: The COV thinks that the program has achieved  |            |
| a good technical balance and a broad variety of         |            |
| expertise in its collection of reviewers.               |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    | Yes        |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments: The program achieves a high level of          |            |
| diversity in all these categories.                      |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      | Yes        |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments: The program as well as NSF is very careful    |            |
| about conflicts of interest.                            |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
|                                                         |            |
| The CoV did not identify any concerns relevant to this  |            |
| area.                                                   |            |
+---------------------------------------------------------+------------+

**A.4 Questions concerning the resulting portfolio of awards under
review.**

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     |               |
| projects supported by the program.                   |               |
|                                                      |               |
| Comments: The COV finds the quality of               |               |
| program-funded research to be very high; the results |               |
| typically appear in the best conferences and         |               |
| journals.                                            |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  | NO            |
| scope of the projects?                               |               |
|                                                      |               |
| Comments: The COV believes NSF should increase the   |               |
| size of its awards. The COV wishes to emphasize that |               |
| it has this belief regardless of the size of NSF     |               |
| budget.                                              |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| High Risk Proposals?                                 |               |
|                                                      |               |
| Comments: The COV finds there are a sufficient       |               |
| number of high-risk proposals in the program.        |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of: Multidisciplinary Proposals?             |               |
|                                                      |               |
| Comments: The COV finds the program has supported    |               |
| several multidisciplinary proposals, especially with |               |
| Signal Processing. The COV considers these           |               |
| quantitatively and qualitatively sufficient.         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Innovative Proposals?                                |               |
|                                                      |               |
| Comments: The COV considers the proposal process     |               |
| highly competitive and most awards innovative.       |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | No            |
| balance of:                                          |               |
|                                                      |               |
| Funding for centers, groups and awards to            |               |
| individuals?                                         |               |
|                                                      |               |
| Comments: The COV has strong reservations in this    |               |
| area. It considers the ERC program within the        |               |
| Engineering Directorate highly beneficial to the     |               |
| engineering and scientific community and the society |               |
| at large. CISE does not have similar programs and    |               |
| the COV highly recommends that CISE consider         |               |
| programs similar to ERCs.                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Awards to new investigators?                         |               |
|                                                      |               |
| Comments: The COV did not find any evidence of a     |               |
| bias in the geographical distribution of Principal   |               |
| Investigators beyond the distribution of the         |               |
| potential pool.                                      |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Geographical distribution of Principal               |               |
| Investigators?                                       |               |
|                                                      |               |
| Comments: The COV considers existing balance         |               |
| sufficient                                           |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Institutional types?                                 |               |
|                                                      |               |
| Comments: The COV considers existing balance         |               |
| sufficient.                                          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Projects that integrate research and education?      |               |
|                                                      |               |
| Comments: The COV considers existing balance         |               |
| sufficient.                                          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance:                                             |               |
|                                                      |               |
| Across disciplines and subdisciplines of the         |               |
| activity and of emerging opportunities?              |               |
|                                                      |               |
| Comments: The COV considers existing balance         |               |
| sufficient.                                          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          | N/A           |
| participation of underrepresented groups?            |               |
|                                                      |               |
| Comments: The COV cannot determine this information  |               |
| based on the review process of random selection      |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      |               |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments: The COV considers existing balance         |               |
| sufficient.                                          |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
|                                                      |               |
| The COV does not have any concerns in these areas.   |               |
+------------------------------------------------------+---------------+

**A.5 Management of the program under review**

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments: **The COV Communication Research (CR) subcommittee has      |
| serious concerns about the organizational structure, the available    |
| resources, and the direction of these resources for the               |
| Communications Research program. These are articulated in the rest of |
| the document.**                                                       |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments: The area of Communications Research has a large number of   |
| emerging research topics. Examples are, proliferation of wireless     |
| local area networks with increased transmission rates and             |
| capabilities, new generation of cellular voice systems, new wireline  |
| technologies such as new generations of digital subscriber lines,     |
| home networking, power line communications, fiber-to-the-home,        |
| optical networking, etc. Basic research as well as product and        |
| service deployment in these areas require a well-trained work force.  |
| The COV CR subcommittee is concerned that the current research and    |
| education support by NSF in these strategic areas may be inadequate.  |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments: Based on a comparison of research budgets under various     |
| programs of the CISE Directorate, the COV CR subcommittee is          |
| concerned that the area of communications, which is strategically     |
| critical to maintaining and broadening world leadership in            |
| electronics, photonics, and computing, may currently be overlooked by |
| the NSF. The subcommittee believes that based on a comparison of the  |
| current relative size of faculty in the field of communications in    |
| the nation's universities with that of those covered in most other    |
| areas under the CISE Directorate, the Communications Research program |
| needs to have a significantly larger budget. The subcommittee         |
| recommends that NSF organize a series of workshops with the           |
| participation of national expertise such as that found in NAE and     |
| IEEE to optimize its organizational structure and funding resources   |
| in this area.                                                         |
|                                                                       |
| [The total budget under the CCR and ANIR clusters of the CISE         |
| Directorate is approximately \$200M in FY2002.]{.underline} [The      |
| Communications Research program budget in FY2002 is \$5.5M, or 2.75%. |
| When one looks at the nation's universities, the number of faculty    |
| pursuing research in the field of communications is usually larger    |
| than 2.75% of the related fields in departments of electrical         |
| engineering and computer science. For example, at UC Berkeley's       |
| Electrical Engineering and Computer Science Department,]{.underline}  |
| there are a total of 136 faculty                                      |
| (www.eecs.bekeley.edu/Faculty/Lists/lists.html). A total of 25 are    |
| emeritus faculty, lecturers, and adjunct faculty. There are 22        |
| faculty whose fields are related to those of the Engineering          |
| Directorate (listed in the next paragraph). Out of the remaining 89   |
| faculty, 15 list Communications as their field (listed below). This   |
| [is a ratio of 16.85%, much larger than 2.75%!]{.underline} When the  |
| comparison is made to the total budget of the CISE Directorate, which |
| is \$519.5M, the ratio is about 1%. **[This is an order of magnitude  |
| less than the ratio of faculty active in the area of Communications   |
| at UCB EECS!]{.underline}**                                           |
|                                                                       |
| Communications faculty at UCB EECS: Anantharam, Brodersen, Gastpar,   |
| Gray, Kahn, Katz, Messerschmitt, Morgan, Nikolic, Rabaey,             |
| Ramchandran, Sahai, Tse, Varaiya, Walrand. Faculty whose research     |
| interests fall under the Engineering Directorate of NSF at UCB EECS:  |
| Attwood, Bokor, Chang-Hasnain, Cheung, Gustafson, Hu, King, Lau,      |
| Lichtenberg, Lieberman, Niknejad, Oldham, Pisano, Sanders, Shank,     |
| Smith, Spanos, Subramnian, Van Duzer, Welch, Whinnery, White.         |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| The COV Communications Research (CR) subcommittee is highly concerned |
| that the area of communications may not be properly represented       |
| within NSF. This concern has been existent in the communications      |
| engineering community for longer than a decade and the current        |
| organizational structure of CISE does not seem to be best suitable in |
| serving this need. Communications technology is a highly critical one |
| and the US leadership of the world in this area is extremely          |
| important to the nation. This leadership position is currently        |
| threatened by other countries and consortiums formed by countries. A  |
| prime example of a lost opportunity is in cellular voice network      |
| technology where a consortium of European companies, under the        |
| support of the European Union, developed a digital cellular voice     |
| communication standard known as GSM. This standard is being used in a |
| large portion of the world today, including some major US markets.    |
| With the loss of the leading industrial research laboratories (Bell   |
| Labs, Bell Communications Research, AT&T Shannon Labs, IBM T. J.      |
| Watson Research Center, etc), the possibility of a similar leadership |
| loss in new telecommunications technologies is a serious concern      |
| among technologists.                                                  |
|                                                                       |
| While this is the case, for NSF to address the fields of              |
| communications only with a small program (Communications Research)    |
| within CCR together with 9 other programs is troublesome and          |
| concerning for the subcommittee. The subcommittee believes the scope  |
| of Communications Research should be increased. In addition, the      |
| subcommittee believes that the services currently supported under the |
| Signal Processing programs of the CCR cluster, as well as the network |
| research aspects of the ANIR cluster have many common aspects. The    |
| subcommittee recommends that a new organizational structure within    |
| the CISE Directorate that enables cooperation and coordination among  |
| organizations responsible from communications, signal processing,     |
| networking, and cryptography be formed and active programs            |
| encompassing all these areas be initiated.                            |
+-----------------------------------------------------------------------+

##### B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.**

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

+-----------------------------------------------------------------------+
| **[B.1 NSF OUTCOME GOAL for PEOPLE:]{.underline}** **Developing "a    |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| Comments: The COV is convinced that the program satisfies the NSF     |
| Outcome Goal for People. As comments and examples of activities and   |
| outcomes, we note                                                     |
|                                                                       |
| [Diversity]{.underline}                                               |
|                                                                       |
| The program supported a proportionally representative number of       |
| female PIs. Examples include                                          |
|                                                                       |
| CCR 9903055, Alexandra Duel-Hallen                                    |
|                                                                       |
| CCR 0093349, Muriel Medard                                            |
|                                                                       |
| CCR 9875582, Maja Bystrom                                             |
|                                                                       |
| CCR 0123409, Tulay Adali                                              |
|                                                                       |
| EIA 9806005, Mary Ann Ingram                                          |
|                                                                       |
| In addition, we note program support for a disabled PI, specifically  |
| in the form of a supplement for travel,                               |
|                                                                       |
| CCR 9876254, Lance Perez.                                             |
|                                                                       |
| The COV notes that during the period of 2000-2002, HBCU and other     |
| minority institution (MI) participation in the submitting proposals   |
| is nonexistent or very small.                                         |
|                                                                       |
| [International Competitiveness]{.underline}                           |
|                                                                       |
| The program is very strong in this area. There are a large number of  |
| examples. We note some below, together with the award number for the  |
| PI from the program.                                                  |
|                                                                       |
| CCR 0118701, Michael Mitzenmacher, 2002 IEEE Information Theory Prize |
| Paper Award                                                           |
|                                                                       |
| CCR 0118842, Michael Zoltowski, 2002 IEEE Signal Processing Society   |
| Technical Achievement Award                                           |
|                                                                       |
| CCR 0238042, Hamid Jafarkhani, 2002 IEEE International Symposium on   |
| Advances in Wireless Communications Best Paper Award                  |
|                                                                       |
| CCR 9903055, Michael Honig, 2002 IEEE Communications Society and      |
| Information Society Award                                             |
|                                                                       |
| CCR 9980616, Toby Berger, 2002 IEEE Information Theory Society Claude |
| E. Shannon Award                                                      |
|                                                                       |
| CCR 0093349, Muriel Medard, 2002 IEEE Leon K. Kirchmayer Prize Paper  |
| Award                                                                 |
|                                                                       |
| CCR 9875511, Madhu Sudan, 2001 IEEE Information Theory Society Prize  |
| Paper Award                                                           |
|                                                                       |
| CCR 0074177, Sergio Verdu, 2001 IEEE Communications Society Leonard   |
| G. Abraham Prize Paper Award                                          |
|                                                                       |
| CCR 0118784, David Tse, 2001 IEEE Communications Society and          |
| Information Theory Society Award                                      |
|                                                                       |
| CCR 9996208, Giorgios B. Giannakis, 2000 IEEE Signal Processing       |
| Society Technical Achievement Award                                   |
|                                                                       |
| CCR 0073505, Michael P. Fitz, 2000 IEEE Communications Society        |
| Leonard G. Abraham Prize Paper Award                                  |
|                                                                       |
| [Globally Engaged]{.underline}                                        |
|                                                                       |
| CCR 0097125, Ilya Dumer requested and received an international       |
| supplement to his grant to collaborate with colleagues in Russia.     |
+-----------------------------------------------------------------------+
| **[B.2 NSF OUTCOME GOAL for IDEAS:]{.underline} Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| **Comments:** The COV notes the following ideas that have had impact  |
| beyond professional conference and journal publications, listed       |
| together with program support for the PI                              |
|                                                                       |
| CCR 9901846, Mohsen Kavehrad. Article in Washington Times, 10/02/02.  |
|                                                                       |
| CCR 0118842, Michael Zoltowski, Zenith Electronics Corporation patent |
| applications                                                          |
|                                                                       |
| CCR 0097114, Jian Li, University of Florida patent applications       |
|                                                                       |
| CCR 9702024, Steven McLaughlin, Chips available through Sanyo and     |
| Calimetrics                                                           |
|                                                                       |
| CCR 0208709, Wojciech Szpankowski, Purdue University patent           |
| applications                                                          |
|                                                                       |
| CCR 9815002, Alexandra Duel-Hallen, North Carolina State University   |
| patent applications                                                   |
|                                                                       |
| CCR 0123409, Tulay Adali, University of Maryland, Baltimore County    |
| patent applications                                                   |
|                                                                       |
| CCR 0098273, Lim Nguyen, University of Nebraska patent applications   |
|                                                                       |
| CCR 9902846, Mohsen Kavehrad, Article in Photonics Technology News,   |
| November 2001                                                         |
|                                                                       |
| CCR 9702024, Steven McLaughlin, ASIC Chip by Calimetrics, Inc.        |
|                                                                       |
| CCR 9902846, Mohsen Kavehrad, Article in New York Times Circuits      |
| Section, August 23, 2001                                              |
|                                                                       |
| EIA 9806005, Mary Ann Ingram, Georgia Institute of Technology patent  |
| applications                                                          |
|                                                                       |
| CCR 9996208, Georgios Giannakis, University of Minnesota patent       |
| applications                                                          |
+-----------------------------------------------------------------------+
| **B.3 OUTCOME GOAL for TOOLS: Providing "broadly accessible,          |
| state-of-the-art and shared research and education tools."**          |
|                                                                       |
| Comments: The COV notes the following tools developed by a Principal  |
| Investigator and is publicly available, again listed together with    |
| the program support                                                   |
|                                                                       |
| CCR 0133521, Ari Trachtenberg. Public domain fast-synching memopad    |
| for the Palm Pilot, available from                                    |
| <http://ipsit.bu.edu/nislab/projects/cpisync/index.htm>. Incorporated |
| into a fast-synching PGP keyserver at                                 |
| [http://keyserver.bu.edu](http://keyserver.bu.edu/), enabling many    |
| users for the PGP public-key cryptosystem.                            |
|                                                                       |
| CCR 9984259, Dan Boneh. First digital signature system that supports  |
| signature aggregation, the compression of a number of signatures on   |
| distinct messages generated by distinct parties into a single         |
| signature for use in certificate chains and in certain secure routing |
| protocols. Open source implementation available from                  |
| <http://crypto.stanford.edu/~dabo>.                                   |
|                                                                       |
| CCR 0093140, Matthew Franklin. The first truly practical              |
| identity-based encryption scheme where the public key can be an       |
| arbitrary string such as the user's email address. Available from     |
| <http://www.stanford.edu/ibe>.                                        |
|                                                                       |
| CCR 9980616, Toby Berger. Open source desktop/laptop video            |
| conferencing software called qvix/cu30 for free noncommercial use.    |
| Available from <http://www.sourceforge.net/projects/cu30>.            |
+-----------------------------------------------------------------------+

##### **C. OTHER TOPICS**

**C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.**

Communications Research (CR) is an area of technology discovery with
deep roots. Beginning well over a century ago, communications via
electrical signals has been, and continues to be, an active area of
research and development. Bell, with his telephone network, and Marconi,
with his experiments in radio, are examples of early researchers in the
wired and wireless communications domain. The field has continued
developing since that time producing technology such as broadband
communications, digital cellular phones, local area networks, both wired
and wireless, high capacity storage devices, the physical layer of
internet and numerous others.

Most early communications systems relied upon analog technology to
reproduce, at the receiver, audio waveforms generated at the
transmitter. Examples include the original telephone system and AM
radio. These systems were optimized by the design of electrical devices,
such as amplifiers and antennas, to try and minimize the distortion of
the signal. Many years after these systems were first deployed,
researchers began to study the benefits of digital communications. It
was soon realized by many (not all) that the process of digitizing
signals into discrete \"zeros and ones\", and then transmitting the
resulting bit stream, would have overwhelming advantages over analog
transmission. (It is interesting to note that one of the very first
electrical transmission systems was the digital telegraph systems using
Morse code.) Although the benefits were apparent many years ago, it took
many decades for all the technology pieces to come together to produce
the impact that we see today. Virtually every new communication system
uses digital transmission and \"advanced\" techniques such as error
correction, data compression, network technology and cryptography.

The original home for Communications Research at the National Science
Foundation was in the Engineering directorate. Even today, the device
side of Communications Research remains in Engineering. This is
appropriate. With the creation of CISE, it was wisely decided that the
system side of Communications Research belonged in the computer area for
several important reasons. The emergence of sophisticated digital signal
processing (i.e., the manipulation of signals with computer technology),
complex integrated digital circuit technology and burgeoning of computer
networks (machine communicating with machine) makes it clear that CR
belongs to CISE.

There is a concern by the COV subcommittee for Communications Research
that important areas of communications systems research have become
seriously fragmented within CISE. It is thought that the research
program would be greatly enhanced by a reorganization that would exploit
the natural synergies that exist between communications, signal
processing, computer networks and cryptography. Under the current
organization, communications research, digital signal processing and
cryptography are within a division that includes research areas with
weak peripheral intersection such as theory of computing, software
engineering, compilers and design automation. Topics such as information
theory, channel coding and cryptography are scattered within
subdivisions. For example, cryptography is supported, to some degree,
within Communications Research, Theory of Computing and Trusted
Computing. It is unclear to us to what extent this important area is
actually supported. Furthermore, the computer networking group is in a
different division of CISE altogether. It seems obvious that there are
stronger connections between communications research and computer
networking than between communications research and most of the computer
areas supported within the CCR division.

The communications research area is a critical component to the strength
and growth of the US economy. The communications area has been important
to the welfare of the nation for over 100 years and is a key component
to the future progress of commerce and individual information needs and
desires. The Internet, which is built from a foundation of physical
layer transmission (i.e., get the bits from here to there or from now to
then) is a permanent fixture. Growth of wireless communications shows no
sign of slowing. At the same time, deregulation of the phone network,
competitive forces in industry and fallout from the \"Internet bubble\"
have seriously degraded the forward looking research being conducted in
industry. For example, Bell Labs, and all its derivatives, have been
drastically diminished. It is critical that the NSF take the lead in
ensuring that this vigorous and vitally important area of research is
fully supported. It is disturbing to the committee that under the
current organization of CISE, communications research is fragmented and
foundering. It would be wise to take prompt action to address this
serious problem at the NSF in a proficient and productive way.

**C.2 Please provide comments as appropriate on the program's
performance in meeting program-specific goals and objectives that are
not covered by the above questions.**

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

We would like to conclude by summarizing our recommendations.

**The relative size of the Communications Research program budget (about
1%) in CISE is currently about an order of magnitude lower than the
relative size of the number of faculty in the field of communications in
the nation's universities (17% at UCB EECS). This is a very heavy
underrepresentation.**

2.The fields of communications, signal processing, and networking have
many commonalities while the organizational structure for these fields
in CISE is fractured. The subcommittee recommends a new organizational
structure that puts these highly relevant fields together.

3.It is unclear where the field of cryptography is supported in NSF.
There are some applications of cryptography supported by the programs
Trusted Computing and Theory of Computing. Communications and networking
applications of cryptography should be supported within the
organizational structure we recommend in item 2 above.

**C.4 Please provide comments on any other issues the COV feels are
relevant.**

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the Communications Research Subcommittee of the CISE Directorate COV

Ender Ayanoglu

Chair

# Signal Processing Systems (SPS)

##### **FY 2003 REPORT TEMPLATE FOR**

#####  **NSF COMMITTEES OF VISITORS (COVs)**

+-----------------------------------------------------------------------+
| **Date of COV: June 5-6, 2003**                                       |
+-----------------------------------------------------------------------+
| **Program/Cluster: Signal Processing Systems**                        |
+-----------------------------------------------------------------------+
| **Division: CCR**                                                     |
+-----------------------------------------------------------------------+
| **Directorate: CISE**                                                 |
+-----------------------------------------------------------------------+
| **Number of actions reviewed by COV**[^32]**:\                        |
| Awards: Declinations: Other:**                                        |
+-----------------------------------------------------------------------+
| **Total number of actions within Program/Cluster/Division during      |
| period being reviewed by COV**[^33]**: Awards: Declinations: Other:** |
+-----------------------------------------------------------------------+
| **Manner in which reviewed actions were selected:**                   |
|                                                                       |
| -   Video conference with Program Director during his site visit to   |
|     Los Angeles                                                       |
|                                                                       |
| -   Review by all subcommittee members of 15 jackets provided by the  |
|     PD as indicative of the range of proposals and resulting actions  |
|     seen by the program                                               |
|                                                                       |
| -   Review by all subcommittee members of an additional 20 jackets    |
|     which were randomly chosen                                        |
|                                                                       |
| -   Review of the documentary materials provided by the Program       |
|     Director and CCR staff                                            |
|                                                                       |
| -   On the second morning additional data and information were        |
|     requested of the Program Director, and the requested material     |
|     were provided immediately                                         |
|                                                                       |
| -   Organization of results by the subcommittee and preparation of    |
|     this report                                                       |
+-----------------------------------------------------------------------+

**A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

**A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures..**

+----------------------------------------------------------+-----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES** | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Is the review mechanism appropriate? (panels, ad hoc     |           |
| reviews, site visits)                                    |           |
|                                                          |           |
| Comments: Process is fair and is executed well. Panels   |           |
| are well chosen with a mix of established and young      |           |
| people over a wide geographical base.                    |           |
+----------------------------------------------------------+-----------+
| Is the review process efficient and effective?           |           |
|                                                          |           |
| Comments: Very effective. Careful reviews are done by    |           |
| selected experts in advance of a panel meeting. The      |           |
| panel system helps get reviews done in a timely fashion. |           |
+----------------------------------------------------------+-----------+
| Are reviews consistent with priorities and criteria      |           |
| stated in the program's solicitations, announcements,    |           |
| and guidelines?                                          |           |
|                                                          |           |
| Comments: Reviewers appear to be following NSF           |           |
| guidelines in their reviews.                             |           |
+----------------------------------------------------------+-----------+
| Do the individual reviews (either mail or panel) provide |           |
| sufficient information for the principal investigator(s) |           |
| to understand the basis for the reviewer's               |           |
| recommendation?                                          |           |
|                                                          |           |
| Comments: All of the reviews examined by the SPS         |           |
| subcommittee justified their reasons and listed proposal |           |
| weaknesses where they saw them. More detail would be     |           |
| better but what was provided on average was enough.      |           |
| There might be a need for some form of score             |           |
| normalization or some reviewer scoring guidelines so     |           |
| that proposal scores can be compared meaningfully across |           |
| programs. SPS reviewers are, as a group, hard to please  |           |
| and grade that way.                                      |           |
+----------------------------------------------------------+-----------+
| Do the panel summaries provide sufficient information    |           |
| for the principal investigator(s) to understand the      |           |
| basis for the panel recommendation?\                     |           |
| Comments: The summaries of strengths and weaknesses      |           |
| seemed to reflect the views expressed by the reviewers   |           |
| and panel members accurately. In general, the level of   |           |
| detail provided seemed sufficient.                       |           |
+----------------------------------------------------------+-----------+
| Is the documentation for recommendations complete, and   |           |
| does the program officer provide sufficient information  |           |
| and justification for her/his recommendation?            |           |
|                                                          |           |
| Comments: From the 30+ jackets examined by the           |           |
| subcommittee it is clear that the program manager drew   |           |
| input from the panel, added comments to it, and also     |           |
| documented his independent assessment of the proposal in |           |
| question.                                                |           |
|                                                          |           |
| The subcommittee observed that the Program Director      |           |
| sometimes held off the rejection of an otherwise         |           |
| fundable proposal pending the possible availability of   |           |
| funds.                                                   |           |
+----------------------------------------------------------+-----------+
| Is the time to decision appropriate?                     |           |
|                                                          |           |
| Comments: The information needed to respond to this was  |           |
| not available to us explicitly for the whole program.    |           |
| The 5-month "time to decision" rule mentioned to us by   |           |
| the Program Manager is an excellent target.              |           |
+----------------------------------------------------------+-----------+
| Discuss issues identified by the COV concerning the      |           |
| quality and effectiveness of the program's use of merit  |           |
| review procedures:                                       |           |
|                                                          |           |
| Exemplary use of well-chosen panels of experts. The      |           |
| Program Director seems to be well supported by the SPS   |           |
| research community.                                      |           |
+----------------------------------------------------------+-----------+

**A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers.**

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       |           |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments: In some cases. However it seems that reviewers |           |
| find it easier to critique technical merit and harder to |           |
| assess broad impact. It should be noted that the impact  |           |
| of some proposals may not be clearly identified until    |           |
| years after submission and successful completion.        |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: In some cases. There is not always an explicit |           |
| discussion of the broader impact.                        |           |
+----------------------------------------------------------+-----------+
| Have the review analyses (Form 7s) addressed whether the |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: There is not always an explicit discussion of  |           |
| the broader impact.                                      |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system.               |           |
|                                                          |           |
| There seems to be a systematic issue here. The "broad    |           |
| impact" concept needs to be made clearer at all levels   |           |
| -- both to the proposal writers and to the reviewers.    |           |
| Neither the PIs nor the reviewers appear to understand   |           |
| exactly what NSF is looking for here.                    |           |
|                                                          |           |
| The Program Director later clarified to the subcommittee |           |
| that the broad impact criteria have only been in force   |           |
| since 1 Oct 02 and therefore would not have been a       |           |
| consideration to the proposal writers or the reviewers   |           |
| during the FY00 to FY02 interval to which the COV was    |           |
| directed.                                                |           |
+----------------------------------------------------------+-----------+

**A.3 Questions concerning the selection of reviewers.**

+---------------------------------------------------------+------------+
| **Selection of Reviewers**                              | **YES,     |
|                                                         | NO,**      |
|                                                         |            |
|                                                         | **DATA NOT |
|                                                         | AVAILABLE, |
|                                                         | or NOT     |
|                                                         | AP         |
|                                                         | PLICABLE** |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       |            |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments: We observed a well balanced group of          |            |
| reviewers -- diverse with respect to geography, rank,   |            |
| and gender.                                             |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            |            |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments: Excellent quality in the choice of reviewers  |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    |            |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments: Excellent. We commend 1) the Program          |            |
| Director's use of a West Coast panel to obtain some     |            |
| geographical diversity and 2) his choice of a mixture   |            |
| in academic rank and station in the panels.             |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      |            |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments: The subcommittee couldn't discern this from   |            |
| the documentation provided. Personal experience from    |            |
| the subcommittee members who have served on panels      |            |
| supports the position that great care is taken to avoid |            |
| conflicts of interest.                                  |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
|                                                         |            |
| Very few of the panel reviewers are from industry, the  |            |
| intelligence community, or the military, all of which   |            |
| have experts in the SPC field. the panels are mostly    |            |
| chosen from the ranks of those who receive money from   |            |
| the NSF.                                                |            |
|                                                         |            |
| The program director is to be commended for his use of  |            |
| foreign signal processing experts on his panels in      |            |
| those cases when he's been able to obtain them.         |            |
+---------------------------------------------------------+------------+

**A.4 Questions concerning the resulting portfolio of awards under
review.**

+---------------------------------------------------------+------------+
| **RESULTING PORTFOLIO OF AWARDS**                       | **APPR     |
|                                                         | OPRIATE,** |
|                                                         |            |
|                                                         | **NOT      |
|                                                         | APPR       |
|                                                         | OPRIATE,** |
|                                                         |            |
|                                                         | **OR DATA  |
|                                                         | NOT        |
|                                                         | A          |
|                                                         | VAILABLE** |
+---------------------------------------------------------+------------+
| Overall quality of the research and/or education        |            |
| projects supported by the program.                      |            |
|                                                         |            |
| Comments: The quality of the proposals the subcommittee |            |
| reviewed was very high. In fact there were many high    |            |
| quality proposals that were not funded because of the   |            |
| program's limited financial resources. The subcommittee |            |
| concurs with the Program Director's emphasis on the     |            |
| systems implications of signal processing rather than   |            |
| the traditional DSP lines of research.                  |            |
+---------------------------------------------------------+------------+
| Are awards appropriate in size and duration for the     |            |
| scope of the projects?                                  |            |
|                                                         |            |
| Comments: The awards are too small to capture the full  |            |
| attention of a principal investigator. Given the        |            |
| program's limited resources, however, we'd not          |            |
| recommend making changes that would lower the number of |            |
| awardees even more. The duration of the awards is       |            |
| reasonable.                                             |            |
+---------------------------------------------------------+------------+
| Does the program portfolio have an appropriate balance  |            |
| of:                                                     |            |
|                                                         |            |
| High Risk Proposals?                                    |            |
|                                                         |            |
| Comments: The Program Director has used his judgement   |            |
| and discretion to fund a good balance of high-risk      |            |
| activities. His use of the SGER avenue for encouraging  |            |
| this type of work is commendable.                       |            |
+---------------------------------------------------------+------------+
| Does the program portfolio have an appropriate balance  |            |
| of: Multidisciplinary Proposals?                        |            |
|                                                         |            |
| Comments: The balance seems fine.                       |            |
+---------------------------------------------------------+------------+
| Does the program portfolio have an appropriate balance  |            |
| of:                                                     |            |
|                                                         |            |
| Innovative Proposals?                                   |            |
|                                                         |            |
| Comments: Virtually all of the proposals funded score   |            |
| very high on the "innovation" scale.                    |            |
+---------------------------------------------------------+------------+
| Does the program portfolio have an appropriate balance  |            |
| of:                                                     |            |
|                                                         |            |
| Funding for centers, groups and awards to individuals?  |            |
|                                                         |            |
| Comments: Most of the funding goes to individual PIs,   |            |
| which the subcommittee believes is appropriate.         |            |
+---------------------------------------------------------+------------+
| Does the program portfolio have an appropriate balance  |            |
| of:                                                     |            |
|                                                         |            |
| Awards to new investigators?                            |            |
|                                                         |            |
| Comments: A strong yes to this. Last year, seven career |            |
| awards were funded as well as 4 or 5 other awards to    |            |
| young investigators. Over the three-year window of      |            |
| review, 22 career awards were made.                     |            |
+---------------------------------------------------------+------------+
| Does the program portfolio have an appropriate balance  |            |
| of:                                                     |            |
|                                                         |            |
| Geographical distribution of Principal Investigators?   |            |
|                                                         |            |
| Comments: The subcommittee found an excellent           |            |
| geographical distribution of awards, but noted a        |            |
| strong, but understandable, emphasis on research        |            |
| universities.                                           |            |
+---------------------------------------------------------+------------+
| Does the program portfolio have an appropriate balance  |            |
| of:                                                     |            |
|                                                         |            |
| Institutional types?                                    |            |
|                                                         |            |
| Comments: Approximately 10% of the awards were given to |            |
| researchers at "lower tier" universities. The           |            |
| subcommittee believes this is an appropriate balance in |            |
| light of the type and quality of proposals received by  |            |
| the program.                                            |            |
+---------------------------------------------------------+------------+
| Does the program portfolio have an appropriate balance  |            |
| of:                                                     |            |
|                                                         |            |
| Projects that integrate research and education?         |            |
|                                                         |            |
| Comments: The subcommittee found that an overwhelming   |            |
| percentage of the standard and continuing awards went   |            |
| to PIs and their graduate students at universities. The |            |
| work conducted by these teams will naturally have a     |            |
| simultaneous research and graduate education component. |            |
| In addition, all of the ITR and career awards feature   |            |
| an educational component, often aimed at the            |            |
| undergraduate level and even lower.                     |            |
+---------------------------------------------------------+------------+
| Does the program portfolio have an appropriate balance: |            |
|                                                         |            |
| Across disciplines and subdisciplines of the activity   |            |
| and of emerging opportunities?                          |            |
|                                                         |            |
| Comments: Over the FY00 to FY03 interval the Program    |            |
| Director appeared to be quite responsive to emerging    |            |
| opportunities in the discipline, forsaking no           |            |
| subdisciplines totally but emphasizing the ones more    |            |
| likely to bear fruit. Particularly commendable was the  |            |
| PD's willingness to fund a small number of "high risk,  |            |
| potentially high reward" efforts under the SGER avenue. |            |
+---------------------------------------------------------+------------+
| Does the program portfolio have appropriate             |            |
| participation of underrepresented groups?               |            |
|                                                         |            |
| Comments: Three of the PD's eight career awards went to |            |
| women, an outstanding ratio. Similarly one of the PD's  |            |
| five EPSCoR proposals was peer-reviewed and funded. The |            |
| subcommittee believes that the program has been very    |            |
| responsive to underrepresented groups.                  |            |
+---------------------------------------------------------+------------+
| Is the program relevant to national priorities, agency  |            |
| mission, relevant fields and other customer needs?      |            |
| Include citations of relevant external reports.         |            |
|                                                         |            |
| Comments: The topic areas supported are well recognized |            |
| as being important. Many of the thrusts supported by    |            |
| SPS were identified as areas of continued and even      |            |
| growing importance in the minutes of the 2002 Defense   |            |
| Applications of Signal Processing Workshop.             |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| the quality of the projects or the balance of the       |            |
| portfolio.                                              |            |
|                                                         |            |
| Given the limited resources provided, the program is    |            |
| managed exceptionally well. The subcommittee members    |            |
| observed that one graduate student and one month of     |            |
| summer support for the PI does not support quality      |            |
| work, or the acceptance of high-risk, high-reward       |            |
| endeavors. We note again that the program has received  |            |
| many high-quality proposals which should have been      |            |
| accepted and would have, had there been sufficient      |            |
| financial resources available. Bio, info, and security  |            |
| areas should also have additional attention from the    |            |
| SPS perspective.                                        |            |
+---------------------------------------------------------+------------+

**A.5 Management of the program under review**

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments: The subcommittee observed --                                |
|                                                                       |
| -   Professional management of the proposal and award process         |
|                                                                       |
| -   Appropriate selection of standard vs. continuing grants,          |
|     particularly for risky efforts                                    |
|                                                                       |
| -   Sensitivity to the full range of diversity issues                 |
|                                                                       |
| -   A willingness to accept risk in speculative areas (even when the  |
|     panels weren't). It is absolutely clear that the Program          |
|     Director's experience in his role has benefited the management of |
|     the program.                                                      |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments: The general strategy appears to be to address emerging      |
| research through the "core program" and to address education through  |
| ITR efforts and the Career awards. The Program Director has responded |
| to emerging trends with his general movement away from classical 1-D  |
| and IMDSP and toward SP in communications and systems-oriented areas  |
| such as sensor networks.                                              |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments: The Program Director seeks recommendations from experts in  |
| the field as to new and emerging areas, and well as to the need to    |
| support more traditional, but still productive, areas. His use of     |
| workshops to gather expert opinion is to be commended.                |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| Concerns                                                              |
|                                                                       |
| -   Too much work for the Program Directors and too little staff to   |
|     ameliorate it.                                                    |
|                                                                       |
| -   Not enough money or time to do site visits to the normal          |
|     \$100K/year PIs.                                                  |
|                                                                       |
| -   Not enough money in general.                                      |
+-----------------------------------------------------------------------+

##### B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.**

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

+-----------------------------------------------------------------------+
| **[B.1 NSF OUTCOME GOAL for PEOPLE:]{.underline}** **Developing "a    |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| Comments: Comments: There has been important independent recognition  |
| of the research and education contributions of SPS awardees. The NSF  |
| has also recognized a number of SPS awards for career-long            |
| contributions to the SPS area and to electrical and computer          |
| engineering in general. Examples of both are listed below \-\--       |
|                                                                       |
| Bob Gray of Stanford, 2002 Presidential Award for Excellence in       |
| Science, Mathematics and Engineering Mentoring (PAESMEM). A bit of    |
| data that suggests the basis for this award \-- He graduated 11 women |
| PhDs in the time between 1985 and 2001.                               |
|                                                                       |
| Vahid Tarokh, NSF Alan T Waterman "Young Scientist" Award.            |
|                                                                       |
| Don Johnson of Rice, IEEE Signal Processing Society's 2000            |
| Meritorious Service Award.                                            |
|                                                                       |
| PP Vaidyanathan of Cal Tech, IEEE Signal Processing Society 2002      |
| Technical Achievement Award.                                          |
|                                                                       |
| Sanjit Mitra of UCSB, IEEE Signal Processing Society's 2001 Society   |
| Award.                                                                |
|                                                                       |
| Vince Poor of Princeton, NSF Director's Award for Distinguished       |
| Teaching Scholars.                                                    |
|                                                                       |
| And more generally 22 NSF CAREER awards, six of them to women.        |
|                                                                       |
| Is the program relevant to national priorities, agency mission,       |
| relevant fields and other customer needs? Include citations of        |
| relevant external reports.                                            |
|                                                                       |
| Comments: The topic areas supported are well recognized as being      |
| important. Many of the thrusts supported by SPS were identified as    |
| areas of continued and even growing important in the minutes of the   |
| 2002 Defense Applications of Signal Processing Workshop.              |
+-----------------------------------------------------------------------+
| **[B.2 NSF OUTCOME GOAL for IDEAS:]{.underline} Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| **Comments:** Much work which will lead to more ideas and inventions; |
| some specific examples                                                |
|                                                                       |
| Bouman and Low, Fluorescence Optical Diffusion Tomography work from   |
| Purdue -- an example of multidisciplinary work with a colleague in    |
| Chemistry. Bouman has partnered with                                  |
|                                                                       |
| Phil Low who is examining methods for detecting tumors using          |
| fluorescein conjugate contrast agents. Bouman et al. have developed a |
| fast and simple method that can image fluorescing tumors obscured by  |
| more than a centimeter of tissue.                                     |
|                                                                       |
| Bresler, Fast algorithms for 3-D medical tomography, from Illinois -- |
| also motivated by a medical problem but applicable to a number of     |
| other fields and applications as well.                                |
|                                                                       |
| -                                                                     |
|                                                                       |
| Paulraj, Wireless Links with Space-time Intermediate Nodes, from      |
| Stanford -- theoretical work that *might* bring a fundamental change  |
| to how commercial wireless networks are built and operated. A number  |
| of important patents and associated start-up companies.               |
+-----------------------------------------------------------------------+
| **B.3 OUTCOME GOAL for TOOLS: Providing "broadly accessible,          |
| state-of-the-art and shared research and education tools."**          |
|                                                                       |
| -   Comments: General education and coursework creation in new and    |
|     evolving fields is a continuing effort by virtually all of the    |
|     SPS awardees.                                                     |
|                                                                       |
| -                                                                     |
|                                                                       |
| -   Connexions -- Rich Baraniuk, a former NYI recipient from Rice     |
|     University, started an educational initiative that has the        |
|     potential to transform how educational materials are developed,   |
|     organized, exchanges among educators, and then delivered to       |
|     students. "Available free of charge, Connexions offers            |
|     custom-tailored, current course material; is adaptable to a       |
|     wide-range of learning styles; and encourages students to explore |
|     the links among courses and disciplines." University-level course |
|     material, and, in particular, material in the SPS area, was the   |
|     first use of the Connexions concept, but others, including the    |
|     Infinity Project listed below, have announced plans to use it as  |
|     well.                                                             |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Connexions has attracted financial backing from Rice and angel    |
|     investors, in addition to a \$1 million gift from Hewlett         |
|     Packard. Rich and his collaborators at Rice now work with         |
|     colleagues from other colleges and universities, and the use of   |
|     the Connexions is expected to grow considerably over the next     |
|     several years.                                                    |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -                                                                     |
|                                                                       |
| -   The Infinity Project \-- "The Infinity Project is a nationally    |
|     recognized partnership between leading research universities,     |
|     industry, government, and K-12 educators to help school districts |
|     incorporate modern engineering and technology in their high       |
|     school curricula." Texas Instruments has provided \$1 million in  |
|     support of the project. Work on the Project has been underway     |
|     during the full three year interval over which this review        |
|     applies. It is now being taught in more than 70 high schools      |
|     nationwide with significant growth planned for the future. It     |
|     teaches engineering and technical concepts using basic examples   |
|     from the field of signal processing systems (SPS). The Project    |
|     has been organized by the signal processing community and, in     |
|     particular, many of the leaders of the project are current and    |
|     past SPS awardees. Specifically, the director of the project is   |
|     Geoffrey Orsak of SMU, and the advisory panel includes Don        |
|     Johnson, Jerry Gibson, Delores Etter, and Leah Jamieson. The      |
|     co-authors of the educational material include Dave Munson, a     |
|     current awardee, and Scott Douglas, a former SPS CAREER awardee.  |
|     Sally Wood, another co-author, served for many years on the NSF's |
|     Committee for Woman, Minorities, and the Handicapped.             |
+-----------------------------------------------------------------------+

##### **C. OTHER TOPICS**

**C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.**

General lack of adequate funding. Program director's initial efforts
toward cultivating work in bio signal processing and cybersecurity
should be expanded if the initial work shows progress. There is an open
question in the subcommittee's mind about Homeland Security and any new
topics that might be suggested by it.

**C.2 Please provide comments as appropriate on the program's
performance in meeting program-specific goals and objectives that are
not covered by the above questions.**

No other goals or objectives identified by the subcommittee

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

-   General lack of funding, particularly in light of growing interest
    in the SPS area

-   If "the second merit review criterion" is to**,** in fact**,** be
    weighted equally with true technical merit, this direction needs to
    be made clearly to all potential PIs and a set of useful guidelines
    provided to them.

**C.4 Please provide comments on any other issues the COV feels are
relevant.**

The management of CCR, and CISE in general, needs to come to the
understanding that the SPS area is every bit as fundamental to the
technical and economic growth of America and the free world as is the
work in computing and networking that it supports.

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

-   Better initial description of the methodology expected of Program
    Directors.

-   Better initial description of relevant administrative details, such
    as how proposal jackets are arranged and expectations regarding
    their contents.

-   Better collaborative tools for accessing NSF information and sharing
    work in the preparation of COV subcommittee reports. There was no
    advanced information that would permit the subcommittee members to
    prepare in the absence of the needed NSF infrastructure.

-   Certain information crucial to the COV review process was provided
    only upon arrival (e.g, the CD). The subcommittee strongly suggests
    that this information be made available sooner (and in a
    transportable format) so that the subcommittee members could prepare
    appropriately before their arrival at NSF.

**C.6** **The Subcommittee's Methodology:**

-   Briefing by PD at the beginning of the review. This briefing was
    held by video teleconference, since the PD was engaged in a site
    visit in Los Angeles that day.

-   Review by all subcommittee members of 15 jackets provided by the PD
    as indicative of the range of proposals and resulting actions seen
    by the program.

-   Review by all subcommittee members of an additional 20 jackets which
    were randomly chosen.

-   Review of the documentary materials provided by the Program Director
    and CCR staff.

-   Request for additional materials (on the second morning), which were
    provided immediately by the PD.

-   Organization of results by the subcommittee and preparation of this
    report.

**C.7** **Longer Term SPS Contributions to Commerce and Society**

Facilitated by the ever improving performance of semiconductors,
academic and industrial research in the SPS area since 1965 has moved
from the lab, to the military, to industrial use, to commercial
applications, and finally to the consumer. A list of these contributions
that have come to prominence over the past three-year reporting interval
include the following:

-   Video teleconferencing, both telephony- and Internet-based

    -   Video and voice compression

    -   Networking

    -   Adaptive echo canceling

-   2 ½ and 3-G cellular phones

-   Advanced hearing aids

-   DVDs and CDs

-   IPODs

-   Commercial network quality speech recognition (e.g., United Airlines
    flight information)

-   Voice over IP for enterprise and carrier-grade use

-   Broadcast HDTV

    -   Modulation theory

    -   Blind and "semi-blind" adaptive equalization

    -   Channel modeling

    -   Compression of audio and video

-   DSL modems

-   Cable internet modems

-   Medical imaging of various types (acoustical, xray, MRI)

-   Digital cameras

And coming soon ...

-   Adaptive dispersion compensation for optical fiber, increasing the
    effective bandwidth of fiber by another factor of 4 to 16, and with
    it the continued decrease in the cost of bandwidth for networking
    applications.

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the Signal Processing Systems \[SPS\]

Ron Schafer

Chair

# Embedded and Hybrid Systems (EHS)

##### FY 2003 REPORT TEMPLATE FOR

#####  NSF COMMITTEES OF VISITORS (COVs)

+-----------------------------------------------------------------------+
| ##### Date of COV: June 5-6^th^, 2003                                 |
+-----------------------------------------------------------------------+
| **Program/Cluster: Embedded and Hybrid Systems**                      |
+-----------------------------------------------------------------------+
| **Division: Computer and Communications Research**                    |
+-----------------------------------------------------------------------+
| **Directorate: Computer and Information Science and Engineering**     |
+-----------------------------------------------------------------------+
| **Number of actions reviewed by COV**[^34]**: Awards: 30              |
| Declinations: 3 Other:**                                              |
+-----------------------------------------------------------------------+
| **Total number of actions within Program/Cluster/Division during      |
| period being reviewed by COV**[^35]**: Awards: Declinations: Other:** |
+-----------------------------------------------------------------------+
| **Manner in which reviewed actions were selected: Since this is a new |
| program, we reviewed all jackets for awards that were made available. |
| We also examined representative "borderline" proposals that were      |
| declined.**                                                           |
+-----------------------------------------------------------------------+

**PART A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures. **Provide comments in the space below the
question**. **Discuss areas of concern in the space provided.**

+----------------------------------------------------------+-----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES** | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Is the review mechanism appropriate? (panels, ad hoc     | Yes       |
| reviews, site visits)                                    |           |
|                                                          |           |
| Comments: While it can be hard to compose good panels in |           |
| new programs, Dr. Gill has done an excellent job in      |           |
| putting together strong panels that (1) have good        |           |
| diversity in terms of gender, ethnicity,                 |           |
| industry/academia mix, senior and rising junior          |           |
| researchers and (2) represent the broad scope of         |           |
| research interests and expertise in the hybrid controls  |           |
| and embedded systems fields.                             |           |
+----------------------------------------------------------+-----------+
| Is the review process efficient and effective?           | Yes       |
|                                                          |           |
| Comments: Despite a large workload (i.e., 96 regular     |           |
| research proposals, 20 career proposals, and 94 ITR      |           |
| proposals), Dr. Gill has done a very good job of meeting |           |
| the GPRA/PART guidelines of processing 80% of the        |           |
| proposals in less than six months.                       |           |
+----------------------------------------------------------+-----------+
| Are reviews consistent with priorities and criteria      | Yes       |
| stated in the program's solicitations, announcements,    |           |
| and guidelines?                                          |           |
|                                                          |           |
| Comments: The EHS program solicitation called for a      |           |
| mixture of fundamental research and experimental         |           |
| validation. The panelists represented a good mix between |           |
| these different facets of the EHS discipline, which was  |           |
| reflected in the reviews for the proposals.              |           |
+----------------------------------------------------------+-----------+
| Do the individual reviews (either mail or panel) provide | Yes       |
| sufficient information for the principal investigator(s) |           |
| to understand the basis for the reviewer's               |           |
| recommendation?                                          |           |
|                                                          |           |
| Comments: A substantial majority of the reviews provided |           |
| in-depth feedback to PIs on the strengths and weaknesses |           |
| of their proposals. In the few cases where the reviews   |           |
| were insufficient, Dr. Gill took this into account when  |           |
| weighting the impact of these reviews on her             |           |
| recommendation for funding the proposals.                |           |
+----------------------------------------------------------+-----------+
| Do the panel summaries provide sufficient information    | **Yes**   |
| for the principal investigator(s) to understand the      |           |
| basis for the panel recommendation?                      |           |
|                                                          |           |
| Comments: In most cases the panel summaries provided     |           |
| in-depth feedback to the PIs. The CoV subcommittee felt  |           |
| that there were cases, however, where it was clear that  |           |
| considerable discussion had taken place amongst the      |           |
| panelists to reach their conclusions and this            |           |
| information was not always captured as thoroughly in the |           |
| summaries as it appeared to have been discussed in the   |           |
| group.                                                   |           |
|                                                          |           |
| The subcommittee is optimistic that the new NSF online   |           |
| system for capturing comments during the panel           |           |
| discussions will help to facilitate more thorough        |           |
| feedback to PIs.                                         |           |
+----------------------------------------------------------+-----------+
| Is the documentation for recommendations complete, and   | Yes       |
| does the program officer provide sufficient information  |           |
| and justification for her/his recommendation?            |           |
|                                                          |           |
| Comments: The review analysis and justification of Dr.   |           |
| Gill is very thorough, particularly for the proposals    |           |
| that were recommended for funding.                       |           |
+----------------------------------------------------------+-----------+
| Is the time to decision appropriate?                     | Yes       |
|                                                          |           |
| Comments: The time to decision was appropriate relative  |           |
| to the GPRA/PART guidelines of processing 80% of the     |           |
| proposals in less than six months.                       |           |
+----------------------------------------------------------+-----------+
| Discuss issues identified by the COV concerning the      |           |
| quality and effectiveness of the program's use of merit  |           |
| review procedures: Based on the CoV subcommittee's       |           |
| review of the EHS jackets, it is clear that Dr. Gill has |           |
| done an exemplary job of ensuring a high quality,        |           |
| thorough, and efficient merit review process.            |           |
+----------------------------------------------------------+-----------+

A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers. **Provide comments in the space below the question.
Discuss issues or concerns in the space provided.**

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **Yes No  |
|                                                          | N/A**     |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       | **Yes**   |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments: Nearly all the individual reviews we examined  |           |
| addressed both intellectual merit and broader impact.    |           |
| Due to Dr. Gill's excellent composition of the panels    |           |
| (which contained a mixture of industry and academic      |           |
| researchers), the panels were in a good position to      |           |
| evaluate both types of contributions.                    |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     | **Yes**   |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments: All of the panel summaries we reviewed in the  |           |
| EHS jackets addressed both merit review criteria.        |           |
+----------------------------------------------------------+-----------+
| Have the *review analyses* (Form 7s) addressed whether   | **Yes**   |
| the proposal contributes to both merit review criteria?  |           |
|                                                          |           |
| Comments: All of the review analyses we reviewed in the  |           |
| EHS jackets addressed both merit review criteria.        |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system. Based on our  |           |
| analysis of the jackets, the EHS research community      |           |
| seems to be firmly committed to evaluating proposals     |           |
| based both on their intellectual merit and their broader |           |
| impacts.                                                 |           |
+----------------------------------------------------------+-----------+

A.3 Questions concerning the selection of reviewers. **Provide comments
in the space below the question. Discuss areas of concern in the space
provided.**

+---------------------------------------------------------+------------+
| Selection of Reviewers                                  | YES , NO,  |
|                                                         |            |
|                                                         | DATA NOT   |
|                                                         | AVAILABLE, |
|                                                         |            |
|                                                         | or NOT     |
|                                                         | APPLICABLE |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       | **Yes**    |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments: All of the EHS jackets we reviewed had at     |            |
| least three reviewers. Most of them had four or five    |            |
| reviewers.                                              |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            | **Yes**    |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments: The panelists selected by Dr. Gill constitute |            |
| many of the top experts in the field, both from         |            |
| industry and from academia.                             |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    | **Yes**    |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments: Dr. Gill did a very good job of including a   |            |
| broad spectrum of reviewers for her panels. In          |            |
| particular, all of the panels we reviewed included      |            |
| several women. She also organized panels with other     |            |
| underrepresented groups, such as African Americans. In  |            |
| addition, her panels had an excellent mixture of        |            |
| reviewers from academia and industry, who possessed a   |            |
| broad coverage of interests and expertise in the EHS    |            |
| field.                                                  |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      | **Yes**    |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments: The CoV subcommittee found no questionable    |            |
| conflict of interest cases in the EHS jackets that were |            |
| examined.                                               |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
+---------------------------------------------------------+------------+

A.4 Questions concerning the resulting portfolio of awards under
review**. Provide comments in the space below the question. Discuss
areas of concern in the space provided.**

+------------------------------------------------+--------+-----------+
| **RESULTING PORTFOLIO OF AWARDS**              | **AP   |           |
|                                                | PROPRI |           |
|                                                | ATE,** |           |
|                                                |        |           |
|                                                | **NOT  |           |
|                                                | AP     |           |
|                                                | PROPRI |           |
|                                                | ATE,** |           |
|                                                |        |           |
|                                                | **OR   |           |
|                                                | DATA   |           |
|                                                | NOT    |           |
|                                                | AVAIL  |           |
|                                                | ABLE** |           |
+------------------------------------------------+--------+-----------+
| Overall quality of the research and/or         | Appro  |           |
| education projects supported by the program.   | priate |           |
|                                                |        |           |
| Comments: The program is supporting very high  |        |           |
| quality fundamental scientific research        |        |           |
| pertaining to embedded and hybrid systems. Due |        |           |
| to the low level of available program funds    |        |           |
| (i.e., \~\$4 million dollars in funding vs. \> |        |           |
| \$70 million in requests), however, the        |        |           |
| program is not yet able to support the same    |        |           |
| level of very high quality experimental        |        |           |
| projects.                                      |        |           |
+------------------------------------------------+--------+-----------+
| Are awards appropriate in size and duration    | Appro  |           |
| for the scope of the projects?                 | priate |           |
|                                                |        |           |
| Comments: Again, due to the low level of       |        |           |
| available program funds relative to requests,  |        |           |
| some awards were not adequate to cover the     |        |           |
| necessary scientific and experimental scope.   |        |           |
| This problem is not due to a failure on the    |        |           |
| part of Dr. Gill, but rather due to the        |        |           |
| significant mismatch between the high level of |        |           |
| interest in EHS from the research community    |        |           |
| vs. the amount of available funds.             |        |           |
+------------------------------------------------+--------+-----------+
| Does the program portfolio have an appropriate | Appro  |           |
| balance of:                                    | priate |           |
|                                                |        |           |
| High Risk Proposals?                           |        |           |
|                                                |        |           |
| Comments: Some of the fundamental research     |        |           |
| projects are prime examples of high-risk, high |        |           |
| payoff efforts. In particular, Dr. Gill        |        |           |
| employed the SGER mechanism to fund the most   |        |           |
| risky research proposal.                       |        |           |
+------------------------------------------------+--------+-----------+
| Does the program portfolio have an appropriate | Appro  |           |
| balance of:                                    | priate |           |
|                                                |        |           |
| Multidisciplinary Proposals?                   |        |           |
|                                                |        |           |
| Comments: One of the primary strengths of the  |        |           |
| EHS program is its inherent focus on combining |        |           |
| the disciplines of hybrid control with         |        |           |
| embedded software.                             |        |           |
+------------------------------------------------+--------+-----------+
| Does the program portfolio have an appropriate |        | Ap        |
| balance of:                                    |        | propriate |
|                                                |        |           |
| Innovative Proposals?                          |        |           |
|                                                |        |           |
| Comments: The majority of the awards in the    |        |           |
| EHS program are highly innovative. Since this  |        |           |
| is a new program area at NSF, there is         |        |           |
| considerable energy and innovation emanating   |        |           |
| from the EHS research community that is        |        |           |
| reflected in the proposals.                    |        |           |
+------------------------------------------------+--------+-----------+
| Does the program portfolio have an appropriate |        | Ap        |
| balance of:                                    |        | propriate |
|                                                |        |           |
| Funding for centers, groups and awards to      |        |           |
| individuals?                                   |        |           |
|                                                |        |           |
| Comments: Many of the EHS jackets the CoV      |        |           |
| subcommittee examined were multi-PI proposals. |        |           |
| In fact, there were 169 PIs on the 96 regular  |        |           |
| research proposals received by the EHS program |        |           |
| in 2002. Due to funding constraints, however,  |        |           |
| the amount of funding for centers and groups   |        |           |
| of PIs was insufficient in many cases.         |        |           |
+------------------------------------------------+--------+-----------+
| Does the program portfolio have an appropriate |        | Ap        |
| balance of:                                    |        | propriate |
|                                                |        |           |
| Awards to new investigators?                   |        |           |
|                                                |        |           |
| Comments: The EHS program selected 8 CAREER    |        |           |
| awards out of a total of 20 proposals          |        |           |
| submitted (and co-funded 2 more with ENG).     |        |           |
| This represented \~20% of the total number of  |        |           |
| awards (43).                                   |        |           |
+------------------------------------------------+--------+-----------+
| Does the program portfolio have an appropriate |        | Ap        |
| balance of:                                    |        | propriate |
|                                                |        |           |
| Geographical distribution of Principal         |        |           |
| Investigators?                                 |        |           |
|                                                |        |           |
| Comments: The 43 awards were distributed       |        |           |
| throughout 22 states, including four EPSCoR    |        |           |
| funded and three EPSCoR co-funded efforts. The |        |           |
| distribution of awards across these 22 states  |        |           |
| was relatively even.                           |        |           |
+------------------------------------------------+--------+-----------+
| Does the program portfolio have an appropriate |        | Ap        |
| balance of:                                    |        | propriate |
|                                                |        |           |
| Institutional types?                           |        |           |
|                                                |        |           |
| Comments: The balance of institutional types   |        |           |
| was appropriate for the research objectives of |        |           |
| the EHS program.                               |        |           |
+------------------------------------------------+--------+-----------+
| Does the program portfolio have an appropriate |        | Ap        |
| balance of:                                    |        | propriate |
|                                                |        |           |
| Projects that integrate research and           |        |           |
| education?                                     |        |           |
|                                                |        |           |
| Comments: Education was one of the key aspects |        |           |
| of broader impacts in the awarded proposals.   |        |           |
+------------------------------------------------+--------+-----------+
| Does the program portfolio have an appropriate |        | Ap        |
| balance:                                       |        | propriate |
|                                                |        |           |
| Across disciplines and subdisciplines of the   |        |           |
| activity and of emerging opportunities?        |        |           |
|                                                |        |           |
| Comments: The EHS portfolio does a superb job  |        |           |
| of representing the fundamental scientific     |        |           |
| subdisciplines of embedded and hybrid systems. |        |           |
| Due to the limited funding, however, the       |        |           |
| experimental and systems subdisciplines are    |        |           |
| underrepresented. These latter subdisciplines  |        |           |
| typically require higher levels of funding to  |        |           |
| achieve significant impact on the field and    |        |           |
| broader impact on industry and the economy     |        |           |
+------------------------------------------------+--------+-----------+
| Does the program portfolio have appropriate    |        | Ap        |
| participation of underrepresented groups?      |        | propriate |
|                                                |        |           |
| Comments: 100% of women who submitted          |        |           |
| proposals to EHS received awards.              |        |           |
+------------------------------------------------+--------+-----------+

+----------------------------------------------------------+-----------+
| Is the program relevant to national priorities, agency   | **Appr    |
| mission, relevant fields and other customer needs?       | opriate** |
| Include citations of relevant external reports.          |           |
|                                                          |           |
| Comments:                                                |           |
|                                                          |           |
| The EHS program is responding to the increased fusion of |           |
| information processing with physical processes that is   |           |
| changing the physical world around us. From toys to      |           |
| aircrafts and from cars to factory robots, computers     |           |
| monitor and control our physical environment. From       |           |
| single-vehicle anti-lock braking control systems to      |           |
| large-scale network-centric combat systems, embedded and |           |
| hybrid systems (EHS) software is playing a greater role  |           |
| in critical infrastructures for such diverse areas as    |           |
| industrial automation, transportation, medical devices,  |           |
| and defense systems.                                     |           |
|                                                          |           |
| EHS software has been neglected as a topic in software,  |           |
| networking, and security research until recently. Viewed |           |
| as an *ad hoc* issue and subordinated to design          |           |
| methodologies for engineered systems, EHS software       |           |
| practice has largely focused on developing small,        |           |
| one-of-a-kind controllers that execute in dedicated      |           |
| computing environments. Proprietary, monolithic, closed  |           |
| real-time operating systems and incomplete tool chains   |           |
| have traditionally dominated EHS computing               |           |
| infrastructure. Research opportunities have been damped  |           |
| in this environment, while the challenges for modern     |           |
| complex engineered systems have increased rapidly in     |           |
| mission- and safety-critical commercial and defense      |           |
| applications. The implications of this disarray are      |           |
| staggering:                                              |           |
|                                                          |           |
| A university and industrial research infrastructure that |           |
| is incapable of supporting next-generation IT-enabled    |           |
| physical systems, particularly large-scale mission- and  |           |
| safety-critical systems                                  |           |
|                                                          |           |
| A lack of comprehensive standards for EHS software,      |           |
| resulting in competitive disadvantage for the US in      |           |
| software control technology and product markets and      |           |
|                                                          |           |
| Exceedingly fragile and vulnerable critical              |           |
| infrastructures, such as air traffic management,         |           |
| supervisory control and data acquisition systems for     |           |
| power grids, and national telecommunications systems.    |           |
|                                                          |           |
| A recent NSF sponsored report[^36] from the National     |           |
| Coordination Office (NCO) Software Design and            |           |
| Productivity (SDP) Coordinating Group concluded that     |           |
| "the tight integration of physical and information       |           |
| processes in embedded systems requires the development   |           |
| of a new systems science, which is simultaneously        |           |
| computational and physical. These advances will          |           |
| ultimately require new educational approaches and        |           |
| project management structures representing radical       |           |
| departures from existing models." The U. S. Department   |           |
| of Defense, whose acquisition programs are routinely     |           |
| plagued by flaws in EHS software, has similarly          |           |
| concluded "There may be some scientific problems which   |           |
| are intrinsic to all military systems that systems       |           |
| developers are not grasping."[^37]                       |           |
|                                                          |           |
| A 1999 study at DARPA[^38] argued that contemporary EHS  |           |
| software is being written without regard to the          |           |
| interaction of the software with the physical world. As  |           |
| a result, substantial effort is spent on extensive       |           |
| manual testing and validation. This procedure is costly, |           |
| error-ridden, and prone to schedule slippage since       |           |
| serious software and systems problems are only           |           |
| discovered late in project lifecycles, e.g., during      |           |
| system integration. Moreover, a recent workshop on       |           |
| Critical Infrastructure Protection (CIP)[^39] has        |           |
| revealed that major gaps exist in the general area of    |           |
| security for EHS software. These gaps have already       |           |
| serious manifestations in areas such as SCADA systems    |           |
| and in networked embedded systems.                       |           |
+----------------------------------------------------------+-----------+

+-----------------------------------------------------------------------+
| **A.5 Management of the program under review. Please comment on:**    |
|                                                                       |
| Management of the program.                                            |
|                                                                       |
| Comments: The management of the EHS program has been very good,       |
| particularly given the circumstances of limited budgets. In           |
| particular, Dr. Gill has shown a good vision for integrating          |
| previously stove-piped communities (e.g., compilers, real-time        |
| scheduling, operating systems, object-oriented design and             |
| programming, and software/hardware co-design) to support the          |
| emergence of the new systems science of embedded and hybrid systems,  |
| which combines information technology with control of physical        |
| properties and devices.                                               |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The EHS program is responding to emerging research and education      |
| trends stemming from the increasingly dominant components of future   |
| physical and engineered computing systems. In particular, the growing |
| demand to connect computers with each other and with sensors and      |
| effectors motivates the need for network-centric systems of           |
| increasing scale and greater precision to better organize collective  |
| interactions and behaviors. As these systems grow, the need grows to  |
| keep application programming relatively independent of the complex    |
| issues of distribution and scale. In addition, systems of national    |
| scale, such as the US air traffic control system, telecommunication   |
| network, and power grid, will of necessity evolve incrementally by    |
| many organizations, using an as yet undefined common engineering      |
| development paradigm.                                                 |
|                                                                       |
| The steady diffusion of EHS software throughout our critical system   |
| infrastructures motivates a significant and sustained national-level  |
| attention to, and investment in, research, education, and engineering |
| of EHS software. Some of the main drivers of this interest are the    |
| following:                                                            |
|                                                                       |
| EHS software is becoming the *universal system integrator* for        |
| physical systems. Its pervasiveness is well illustrated by the        |
| following facts: (a) the total shipment of microprocessor units and   |
| micro control units in 1997 was over 4.4 billion units, and of this   |
| about 98% related to embedded applications[^40] and (b) between 1994  |
| and 2004 the need for EHS software developers is expected to increase |
| 10-fold.[^41]                                                         |
|                                                                       |
| EHS computing represents a disruptive technology for established      |
| industries, e.g., 80-90% of all innovations in the automotive         |
| industry are now based on EHS computing.[^42] In particular, the      |
| expected impact of drive-by-wire technologies is revolutionary.[^43]  |
|                                                                       |
| New and pressing requirements for advances in EHS technology derive   |
| from the need for protection of our critical infrastructure.          |
| According to a recent report from the President of the US: "The need  |
| for \[Information and Communication Critical Infrastructure           |
| Protection (CIP)\] creates a zone of shared responsibility and        |
| cooperation among industry, government, and *academia*. If we are to  |
| retain and build upon the competitive edge information technology has |
| given us, we need to work together on CIP R&D and in other pursuits   |
| to substantially improve the trustworthiness of our information       |
| systems and networks."[^44] The pervasiveness of EHS software in all  |
| industries has created an invisible IT infrastructure, which has      |
| tremendous impact on the security of our critical infrastructure.     |
| Early indications of potential problems with SCADA systems^8^ show    |
| that we must revamp our approach to embedded system development and   |
| security.                                                             |
+-----------------------------------------------------------------------+
| EHS computing presents a vitally important opportunity and challenge  |
| for national defense and homeland security. From avionics systems to  |
| smart weapons and bio/chem detectors, embedded information processing |
| is the primary source for superiority in combat and homeland security |
| systems. The new wave of inexpensive MEMS-based sensors and actuators |
| and the continued progress in computing and communication technology  |
| will further accelerate this trend. Defense and homeland security     |
| systems will become increasingly "information rich", where embedded   |
| monitoring, control and diagnostic functions penetrate deeper and     |
| with smaller granularity in physical component structures.Program     |
| planning and prioritization process (internal and external) that      |
| guided the development of the portfolio under review.                 |
|                                                                       |
| Comments: The planning and prioritization for the EHS program         |
| portfolio has been guided by a number of workshops and collaborative  |
| efforts between NSF, NCO, DARPA, other US Government agencies and     |
| services, as well as commercial industry, over the past two years. In |
| particular, it has been guided by the findings of NSF-sponsored       |
| workshops[^45]^,^[^46]^,^[^47]^,^[^48]^,^[^49]^,^[^50] and            |
| IEEE-sponsored conferences[^51]^,^[^52], contributions from research  |
| groups[^53] in the joint NSF/DARPA collaborative National             |
| Experimentation Platform from Hybrid and Embedded Systems (NEPHEST)   |
| effort and consultation with leading researchers and experts from     |
| major EHS end-user industries (such as Boeing, Ford, GM,              |
| Lockheed-Martin, and Raytheon).                                       |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| The CoV subcommittee identified the following key areas of concern    |
| for the EHS program:                                                  |
|                                                                       |
| Lack of funds to support the necessary level of highly innovative     |
| work in the field                                                     |
|                                                                       |
| Lack of adequate administrative support                               |
|                                                                       |
| Lack of travel funds, which make it hard to manage the program via    |
| site visits and periodic technical exchanges                          |
+-----------------------------------------------------------------------+

PART B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.**

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

To promote the progress of science.

To advance national health, prosperity, and welfare.

To secure the national defense.

And for other purposes.

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

+-----------------------------------------------------------------------+
| **B.1 [NSF OUTCOME GOAL for PEOPLE]{.underline}: Developing "a        |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| Comments: Two tangible examples of the contribution of the EHS        |
| program to diversified preparation of the scientific and engineering  |
| workforce are                                                         |
|                                                                       |
| The Summer Internship Program in Hybrid and Embedded Software         |
| (SIPHER), which is currently run by Vanderbilt University and         |
| University of California Berkeley as part of their cooperative        |
| research agreement (CCR-0225610) "Foundations of Hybrid and Embedded  |
| Software Systems." The SIPHER program is organized around the         |
| outreach effort toward HBCUs with a critical mass of minority         |
| students and community colleges both in the South and in California.  |
| Dr. Brian Williams, Assistant Dean for Student Affairs at Vanderbilt  |
| University, is establishing linkages to schools like Kisk University, |
| Tennessee State University, University of Alabama-Birmingham,         |
| Morehouse College, Spelman College, Florida State University, Florida |
| A&M University and others. The SIPHER project is profoundly changing  |
| the current curriculums in embedded and hybrid systems at Vanderbilt  |
| University and UC Berkeley.                                           |
|                                                                       |
| The 10 CAREER awards, which span an important range of EHS topics and |
| geographical regions in the US. Since the EHS area is historically    |
| underrepresented at NSF, the CAREER awards in this area are           |
| particularly important to help to foster a diverse, internationally   |
| competitive, and globally engaged scientific talent pool.             |
|                                                                       |
| Dr. Gill has been instrumental in fostering international             |
| cooperations between scientists and engineers in the US and EU. These |
| cooperations have included (1) the US/EU joint workshops on Critical  |
| Infrastructure Protection in Lansdowne VA (USA) and Dusseldorf        |
| Germany and (2) a review of the EU ARTIST program, which was held in  |
| Grenoble France in August 2002 and which focused on collaborations    |
| between US and EU EHS researchers from academia and industry.         |
+-----------------------------------------------------------------------+
| **B.2 [NSF OUTCOME GOAL for IDEAS]{.underline}: Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| Comments: **Embedded and hybrid systems (EHS), which integrate with   |
| and respond appropriately to both physical world and human            |
| constraints, make demands on systems in new ways. The comfort zone    |
| for building conventional software intensive systems currently lies   |
| in stable, well understood, and predictable environments, where human |
| operators most often bridge the gap between the deficiencies of the   |
| original software and the realities of the operating environment.     |
| Staying within this comfort zone is rarely possible for EHS           |
| environments. Instead, software intensive EHS contexts are subject to |
| rapidly changing environments, new sets of failure modes and fault    |
| conditions, and more stringent safety and performance constraints,    |
| which render obsolete** **a substantial part** **of what we know      |
| regarding how to construct working software systems. The following    |
| are some of the promising focus areas funded by the NSF EHS program   |
| that can lead to the breakthrough technologies and approaches for     |
| fieldable quality EHS applications that behave dependably and         |
| correctly under embedded and stressed environments:**                 |
|                                                                       |
| ***Adaptive and reflective middleware*** as a means to develop a      |
| general and reusable capability for constructing EHS applications     |
| that appropriately deliver required quality of service while          |
| responding to rapid changes in the environment                        |
|                                                                       |
| ***Mixed computation*** (e.g., hard/soft real-time and anytime        |
| algorithms) as a means to support the coexistence of EHS capabilities |
| with both stringent and loose requirements within the common extended |
| platforms for the system as a whole                                   |
|                                                                       |
| ***Fault tolerant*** ***embedded systems*** as a means of handling    |
| the extended, unattended lifetime for software intensive EHS          |
| applications subject to both rugged environments and malicious        |
| behavior                                                              |
|                                                                       |
| ***Hybrid (i.e., multi-modal and coordinated) and distributed         |
| control*** as a means of providing coordinated and non-interruptible  |
| EHS behavior and                                                      |
|                                                                       |
| ***High confidence and validated behavior*** as a necessary precursor |
| for applying semi-supervised software intensive EHS applications to   |
| safety- and mission-critical tasks.                                   |
|                                                                       |
| Although the NSF EHS program is still in its infancy, the EHS         |
| research community is focusing on many of the hardest challenges in   |
| computer and systems science today. During the past two years,        |
| increases in innovations and interest in the EHS research community   |
| is shown clearly by the following indicators:                         |
|                                                                       |
| **Special issues of various aspects of EHS software development in    |
| highly selective journals, such as Proceedings of the IEEE**[^54]     |
| **and IEEE Computer**[^55]**.**                                       |
|                                                                       |
| **New archival publication forums**[^56]**^,^**[^57]**.**             |
|                                                                       |
| **National studies and research direction meetings, such as the NRC   |
| Study**[^58]**, ITR&D Coordinating Group                              |
| Reports**[^59]**^,^**[^60]**, NSF/OSTP Workshop on Innovative         |
| Information Technology for Critical Infrastructure Protection^11^     |
| NSF/ACM Workshop on Embedded Systems**[^61]**, and IEEE-sponsored     |
| conferences**^12,13^**.**                                             |
|                                                                       |
| **New research centers and teaching programs at leading universities, |
| such as UC Berkeley, Carnegie Mellon University, University of        |
| Pennsylvania, Georgia Institute of Technology, Vanderbilt University, |
| University of Virginia, University of Michigan, University of         |
| California, Irvine, and others.**                                     |
|                                                                       |
| **Tremendous increase in the number of high quality proposals to the  |
| EHS program of NSF and to the NSF ITR program.**                      |
|                                                                       |
| **A particularly encouraging sign is the strong industry              |
| participation and interest in Federal EHS R&D programs by major US    |
| companies, such as BBN Technologies, Boeing, Ford, GM, Honeywell,**   |
| **Lockheed-Martin, Raytheon, and others.**                            |
+-----------------------------------------------------------------------+
| **B.3 [OUTCOME GOAL for TOOLS:]{.underline} Providing "broadly        |
| accessible, state-of-the-art and shared research and education        |
| tools."**                                                             |
|                                                                       |
| Comments: Although these breakthrough technology ideas and concepts   |
| discussed above are necessary to allow us to move toward constructing |
| high quality EHS technologies, they are not sufficient *in isolation* |
| to field EHS applications with these capabilities rapidly and cost    |
| effectively. To address this problem, new research and education      |
| tools are needed to develop, validate, and deploy EHS applications. A |
| key element of this research agenda is *system construction by        |
| composition* (rather than exclusively by manual organic development)  |
| that takes into account the unique nature of composition for EHS      |
| environments. The following are some of the state-of-the-art tools    |
| that are emerging to support the new software discipline that is      |
| appropriate for EHS environments:                                     |
|                                                                       |
| ***Multi-faceted software composition*** tools that support the       |
| unified design and construction of the many distinct elements of the  |
| total requirements, such that the end result operates over a single,  |
| common shared platform. Examples include the FLEX Real-time Java      |
| compiler from Rinard at MIT (CCR-0209075 and EHS-029211), the         |
| MicroQoSCORBA configurable object request broker by Bakken at         |
| Washington State University (EHS-0209211), the FACET aspect-oriented  |
| event service by Cytron at Washington University St. Louis            |
| (OSC-0203869), and the VEST aspect-oriented embedded systems toolkit  |
| by Stankovic at University of Virginia (OSC-0098269 and EHS-0208769). |
|                                                                       |
| ***Model-based embedded software*** as a means to (1) elevate the     |
| system development abstraction level into the domain space and (2)    |
| provide a more automated and formal basis for designs and tools that  |
| evaluate designs. Examples include the Generic Modeling Environment   |
| (GME) by Karsai and Sztipanovits at Vanderbilt University             |
| (CCR-0225610), the Ptolemy computational modeling toolkit by Lee at   |
| University of California Berkeley (CCR-00225610), the AIRES embedded  |
| systems modeling toolkit by Shin at Michigan University               |
| (OSC-0216977), and the Cadena model-checking analyzer for component   |
| middleware by Hatcliff and Dwyer at Kansas State University           |
| (CCR-0306607).                                                        |
|                                                                       |
| ***Domain-specific frameworks*** for composing and customizing        |
| components as a means of composing components of significant size and |
| appropriate shape to lend meaning to development-by-integration.      |
| Examples include the ACE and TAO real-time middleware frameworks by   |
| Schmidt at Vanderbilt University (NCR-9628218 and CCR-0312859), the   |
| Kokyu dynamic scheduling framework by Gill and Schmidt at Washington  |
| University St. Louis (NCR-9714698), the TMO time-triggered            |
| message-triggered middleware framework by Kim at University of        |
| California Irvine (NGS-9975053) and the MEAD fault tolerant           |
| middleware toolkit by Narasimhan at CMU (EHS-0238381).                |
|                                                                       |
| ***System/software co-design environments*** as a means of meeting    |
| the dual requirements emanating from a hybrid and embedded system as  |
| both a software construct and as a operating entity with real,        |
| observable behavior and characteristics. Examples include the         |
| EXPRESSION system-on-a-chip compiler by Dutt and Gupta at the         |
| University of California Irvine (MIP-9708067) and the CHARON hybrid   |
| systems toolkit from Alur, Lee, and Pappas at the University of       |
| Pennsylvania (EHS-0121431 and EHS-0209024).                           |
|                                                                       |
| These tools are helping to address EHS composition capabilities in    |
| concert with system-wide property challenges to move away from        |
| today's *ad hoc*, overly rigid, costly, and error-prone EHS designs   |
| toward future adaptive, flexible designs that run safely and          |
| accurately. Only when we meet the dual challenges posed by stringent  |
| operating characteristics and a rigorous but easily applied and cost  |
| effectively repeatable engineering methodology, will we be positioned |
| to meet the societal demands and the opportunities posed by the       |
| hardware improvements that are enabling next-generation EHS tools and |
| applications.                                                         |
+-----------------------------------------------------------------------+

#####  PART C. OTHER TOPICS

C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.

**The key gap at this point is to integrate key experimental systems
research in the area of embedded and hybrid systems with the excellent
fundamental scientific foundation that's being supported in the current
program.**

C.2 Please provide comments as appropriate on the program's performance
in meeting program-specific goals and objectives that are not covered by
the above questions.

**The amount of funding for the EHS program relative to other programs
in the C-CR division seems to be very low, particularly given large
amount of interest in the program.**

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

**Given the high proposal workload for the EHS program, the CoV
subcommittee recommends that NSF increase the administrative support
staff in order to ensure that Dr. Gill has adequate time to manage the
proposal review process, as well as administer her existing projects and
plan for new initiatives.**

C.4 Please provide comments on any other issues the COV feels are
relevant.

**EHS research has historically been the "poor stepchild" of information
technology research, with concerns scattered across the computer and
information technology disciplines: software engineering, operating
systems, compilers, computer architecture, design automation,
networking,** **modeling, and simulation. As a result of recent advances
stemming from increased Federal and industrial investment, it is
becoming a vibrant, rapidly growing field with a tremendous application
pull. The significant changes achieved during the past several years are
not yet self-sustaining, however. Without continued significant Federal
investment in fundamentals and without concerted effort by the
stakeholder industry in technology transfer and technology exploitation,
this emerging new area may loose momentum and fall back to old practices
that had proved to be ineffective and technically inferior.**

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format, and report template.**

It would have been particularly helpful to have been given more
information on the process of the CoV meeting and the form/structure of
the CoV subcommittee reports before the meeting itself.

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the C-CR CoV Subcommittee on Embedded and Hybrid Systems

Douglas C. Schmidt

Chair

# Trusted Computing (TC) 

##### **FY 2003 REPORT TEMPLATE FOR**

#####  **NSF COMMITTEES OF VISITORS (COVs)**

+-----------------------------------------------------------------------+
| **Date of COV: June 5-6, 2003**                                       |
+-----------------------------------------------------------------------+
| **Program/Cluster: Trusted Computing**                                |
+-----------------------------------------------------------------------+
| **Division: CCR**                                                     |
+-----------------------------------------------------------------------+
| **Directorate: CISE**                                                 |
+-----------------------------------------------------------------------+
| **Number of actions reviewed by COV**[^62]**:\                        |
| Awards: 38 Declinations: 102 Other:**                                 |
+-----------------------------------------------------------------------+
| **Total number of actions within Program/Cluster/Division during      |
| period being reviewed by COV**[^63]**: Awards: 10 Declinations: 7     |
| Other:**                                                              |
+-----------------------------------------------------------------------+
| **Manner in which reviewed actions were selected:\                    |
| **All of the initial year's awards and their distribution across      |
| technical areas are described in the annual report so this data will  |
| not be repeated in this report.                                       |
|                                                                       |
| The reviewed actions were those selected by PO Carl Landwehr, plus    |
| one additionally requested by the COV. After considering the entire   |
| list of proposals and awards for this program, this selective         |
| strategy was deemed to provide an adequate sample.                    |
+-----------------------------------------------------------------------+

**A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

**Program Overview:**

The trusted program is concerned with how to assure that programs and
systems of programs running on hardware platforms behave as intended
regardless of attacks and failures. While this concern provides a very
broad technical scope, which could overlap with many other concerns in
CISE, the requirement that the work address potential malicious acts
provides an important focus. The Trusted Computing Program is a keystone
of the NSF Cyber Trust Initiative, which also incorporates Network
Security, Data and Applications Security, and Security in Embedded and
Control Systems. Trusted Computing was planned in FY2001 in part to
respond to a recognized national shortage of faculty qualified to
conduct research and teach in this area, and a consequent shortage of
graduates. It was announced on September 5, 2002. Subsequent events have
heightened its relevance, visibility, and interest.

The passage of P.L. 107-305, the Cyber Security Research and
Development, authorized over \$900 million to be spent by NSF and NIST
in a five year period, clearly demonstrating the national concern with
this issue. The COV believes that the existing Trusted Computing
Program, especially as it is currently managed by Program Officer Dr.
Carl Landwehr, is an excellent foundation for implementing the Cyber
Security Research and Development plan. The COV strongly encourages the
NSF to view the Trusted Computing Program as an exemplary cornerstone
for building the larger, broader opportunity that the Cyber Security
Research and Development initiative represents.

**A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures..**

+----------------------------------------------------------+---+----------+
| QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES     | * |          |
|                                                          | * |          |
|                                                          | Y |          |
|                                                          | E |          |
|                                                          | S |          |
|                                                          | , |          |
|                                                          | N |          |
|                                                          | O |          |
|                                                          | , |          |
|                                                          | * |          |
|                                                          | * |          |
|                                                          |   |          |
|                                                          | * |          |
|                                                          | * |          |
|                                                          | D |          |
|                                                          | A |          |
|                                                          | T |          |
|                                                          | A |          |
|                                                          | N |          |
|                                                          | O |          |
|                                                          | T |          |
|                                                          | A |          |
|                                                          | V |          |
|                                                          | A |          |
|                                                          | I |          |
|                                                          | L |          |
|                                                          | A |          |
|                                                          | B |          |
|                                                          | L |          |
|                                                          | E |          |
|                                                          | , |          |
|                                                          | o |          |
|                                                          | r |          |
|                                                          | N |          |
|                                                          | O |          |
|                                                          | T |          |
|                                                          | A |          |
|                                                          | P |          |
|                                                          | P |          |
|                                                          | L |          |
|                                                          | I |          |
|                                                          | C |          |
|                                                          | A |          |
|                                                          | B |          |
|                                                          | L |          |
|                                                          | E |          |
|                                                          | * |          |
|                                                          | * |          |
+----------------------------------------------------------+---+----------+
| Is the review mechanism appropriate? (panels, ad hoc     |   |          |
| reviews, site visits)                                    |   |          |
|                                                          |   |          |
| Comments: NSF review process has been fine-tuned over    |   |          |
| the years and appears to be appropriate for the purposes |   |          |
| of proposal evaluation and award in this program.        |   |          |
+----------------------------------------------------------+---+----------+
| Is the review process efficient and effective?           |   |          |
|                                                          |   |          |
| Comments: The process is effective but the workload on   |   |          |
| the PO and the expectations on panelists are reaching    |   |          |
| criticality. We expect this to be an increasing problem  |   |          |
| in the future.                                           |   |          |
+----------------------------------------------------------+---+----------+
| Are reviews consistent with priorities and criteria      |   |          |
| stated in the program's solicitations, announcements,    |   |          |
| and guidelines?                                          |   |          |
|                                                          |   |          |
| Comments: The COV was impressed by the detail and        |   |          |
| helpfulness of the reviews. The panelists' reviews were  |   |          |
| deemed consistent with the stated program priorities and |   |          |
| criteria.                                                |   |          |
+----------------------------------------------------------+---+----------+
| Do the individual reviews (either mail or panel) provide |   |          |
| sufficient information for the principal investigator(s) |   |          |
| to understand the basis for the reviewer's               |   |          |
| recommendation?                                          |   |          |
|                                                          |   |          |
| Comments:                                                |   |          |
+----------------------------------------------------------+---+----------+
| Do the panel summaries provide sufficient information    |   |          |
| for the principal investigator(s) to understand the      |   |          |
| basis for the panel recommendation?\                     |   |          |
| Comments:                                                |   |          |
+----------------------------------------------------------+---+----------+
| Is the documentation for recommendations complete, and   |   |          |
| does the program officer provide sufficient information  |   |          |
| and justification for her/his recommendation?            |   |          |
|                                                          |   |          |
| Comments: We found several instances in which the PO was |   |          |
| required to use considerable judgment in the face of     |   |          |
| divergent reviews. In each of these cases, the reviews   |   |          |
| were individually sufficiently detailed to allow the PO  |   |          |
| to make a sensible final recommendation. The Program     |   |          |
| Officer, due to his significant domain expertise, was    |   |          |
| able to clearly document decisions that were based on    |   |          |
| programmatic and technical factors                       |   |          |
+----------------------------------------------------------+---+----------+
| Is the time to decision appropriate?                     |   |          |
|                                                          |   |          |
| Comments:                                                |   |          |
|                                                          |   |          |
| The COV understands that 6 months is the target timeline |   |          |
| for making decisions at NSF and that the program is      |   |          |
| meeting that goal. However, it would be good for the     |   |          |
| community if the NSF review process could accelerate the |   |          |
| schedule in any way possible. Note, the 6 month goal     |   |          |
| does not seem to have changed in over 20 years despite   |   |          |
| FastLane, etc.!!                                         |   |          |
+----------------------------------------------------------+---+----------+
| Discuss issues identified by the COV concerning the      |   |          |
| quality and effectiveness of the program's use of merit  |   |          |
| review procedures:                                       |   |          |
|                                                          |   |          |
| The merit review procedures are working very well,       |   |          |
| especially under the skillful implementation of them by  |   |          |
| Carl Landwehr.                                           |   |          |
+----------------------------------------------------------+---+----------+

**A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers.**

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       |           |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments:                                                |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments:                                                |           |
+----------------------------------------------------------+-----------+
| Have the review analyses (Form 7s) addressed whether the |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments:                                                |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system.               |           |
|                                                          |           |
| The reviewers addressed both criteria thoroughly in the  |           |
| sample of the 140 proposals that COV inspected. Even in  |           |
| cases where the individual panelist reviews were light   |           |
| on one of the criteria, the panel summary addressed both |           |
| adequately. The program officer addressed both criteria  |           |
| in the Form 7s on the inspected awards, although the     |           |
| criteria are not always explicitly broken out.           |           |
|                                                          |           |
| The COV noted that the program is new and does not have  |           |
| inbreeding in the panelists or awardee pool, as some     |           |
| other, older programs might have.                        |           |
+----------------------------------------------------------+-----------+

**A.3 Questions concerning the selection of reviewers.**

+---------------------------------------------------------+------------+
| **Selection of Reviewers**                              | **YES,     |
|                                                         | NO,**      |
|                                                         |            |
|                                                         | **DATA NOT |
|                                                         | AVAILABLE, |
|                                                         | or NOT     |
|                                                         | AP         |
|                                                         | PLICABLE** |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       |            |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments: See Comments Below                            |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            |            |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments: See Comments Below                            |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    |            |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments: See Comments Below                            |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      |            |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments: The evidence shows that Conflicts of Interest |            |
| were identified and dealt with properly.                |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
|                                                         |            |
| The COV commends the Program Officer for his background |            |
| and experience in this area and for his selection of    |            |
| panelists and reviewers. This has contributed           |            |
| significantly to the quality of the review process in   |            |
| this program and the selection of the awards.           |            |
|                                                         |            |
| Given where the expertise in trusted computing lies     |            |
| today, there was an appropriate balance among reviewer  |            |
| characteristics such as type of institution, geography, |            |
| and underrepresented groups.                            |            |
+---------------------------------------------------------+------------+

**A.4 Questions concerning the resulting portfolio of awards under
review.**

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     |               |
| projects supported by the program.                   |               |
|                                                      |               |
| Comments: On the whole, the supported projects show  |               |
| a significant awareness of computer systems and      |               |
| network issues that are not generally seen in other  |               |
| basic research programs.                             |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  |               |
| scope of the projects?                               |               |
|                                                      |               |
| Comments: The selected awards are appropriate given  |               |
| the submitted proposal pool and the program's        |               |
| budget, but some projects could not be adequately    |               |
| funded because of budget limitations. The COV        |               |
| believes that in order to sustain national           |               |
| leadership in this area, the Cyber Trust             |               |
| programmatic vision should be implemented to augment |               |
| this program's budget.                               |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| High Risk Proposals?                                 |               |
|                                                      |               |
| Comments: Award duration is appropriate for the size |               |
| of the currently funded projects.                    |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of: Multidisciplinary Proposals?             |               |
|                                                      |               |
| Comments: The COV notes that no truly                |               |
| multidisciplinary proposals were submitted and       |               |
| therefore none were awarded.                         |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Innovative Proposals?                                |               |
|                                                      |               |
| Comments: The Program Officer selected the most      |               |
| promising high risk/high payoff and innovative       |               |
| proposals wisely.                                    |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Funding for centers, groups and awards to            |               |
| individuals?                                         |               |
|                                                      |               |
| Comments: Most proposals were for one or two PI      |               |
| efforts and the awards reflect that. The             |               |
| establishment of a quota for an appropriate balance  |               |
| among centers, groups and individuals is inherently  |               |
| misguided because the proposals should be considered |               |
| on the basis of technical merit and broader impact,  |               |
| rather than any quota fulfillment                    |               |
+------------------------------------------------------+---------------+
|                                                      |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Geographical distribution of Principal               |               |
| Investigators?                                       |               |
|                                                      |               |
| Comments: Proposals were considered on merit and the |               |
| resulting geographical distribution is adequate.     |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Institutional types?                                 |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| Projects that integrate research and education?      |               |
|                                                      |               |
| Comments: Most awards were made to Ph.D. granting    |               |
| educational institutions. This reflects the          |               |
| submitted proposals' originating organizations.      |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance:                                             |               |
|                                                      |               |
| Across disciplines and subdisciplines of the         |               |
| activity and of emerging opportunities?              |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          |               |
| participation of underrepresented groups?            |               |
|                                                      |               |
| Comments: Within the limitations of what was         |               |
| proposed, the awards reflect an appropriate balance  |               |
| of projects integrating research and education,      |               |
| representing the breadth of Trusted Computing, and   |               |
| providing participation of underrepresented groups.  |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      |               |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments: The COV strongly believes that this        |               |
| program is absolutely relevant and necessary to      |               |
| national needs and priorities.                       |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
+------------------------------------------------------+---------------+

**A.5 Management of the program under review**

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments: The Program Officer did a superb job in defining the        |
| program and allocating resources.                                     |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments: With respect to research and education, the program is      |
| clearly responsive within scope of the available funding and the      |
| submitted proposals.                                                  |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments:                                                             |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
+-----------------------------------------------------------------------+

B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

**B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.**

+-----------------------------------------------------------------------+
| **[B.1 NSF OUTCOME GOAL for PEOPLE:]{.underline}** **Developing "a    |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| Comments: Because the program is only one year old, it is premature   |
| to attempt to assess the success of the individual researchers.       |
| Considering that this area is growing and attracting researchers from |
| other areas, some strategy for giving newcomers practical experience  |
| with trusted computing issues in a production environment would be    |
| good to explore. This would also have impact across the breadth of    |
| the CS and CE curricula, which is urgently needed.                    |
+-----------------------------------------------------------------------+
| **[B.2 NSF OUTCOME GOAL for IDEAS:]{.underline} Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| **Comments:** The program is supporting a spectrum of projects        |
| ranging from basic science through to applications relevant to        |
| current national priorities. Connections with real systems and real   |
| needs should be sustained as suggested above.                         |
+-----------------------------------------------------------------------+
| **B.3 OUTCOME GOAL for TOOLS: Providing "broadly accessible,          |
| state-of-the-art and shared research and education tools."**          |
|                                                                       |
| **Comments:** This program is only one year old. Tools typically      |
| follow basic research by a few years. However, tools are fundamental  |
| to utilizing basic research results and thereby improving security.   |
| The currently funded PI's are entirely capable of developing tools    |
| that build on their research.                                         |
+-----------------------------------------------------------------------+

**B.4 ANTICIPATED OUTCOME FROM THE PROGRAM**

Certain expectations can already be drawn from the nature of the FY2002
awards in the Trusted Computing Program. Some promising research
directions funded in FY2002 are:

-   dynamic schemes for detecting user departure and requiring
    subsequent re- authentication;

-   injected hardware faults to attack virtual machines;

-   virtual machine based logging;

-   distributed systems security through automated information flow
    management.

#####  **C. OTHER TOPICS**

**C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.**

**C.2 Please provide comments as appropriate on the program's
performance in meeting program-specific goals and objectives that are
not covered by the above questions.**

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

**C.4 Please provide comments on any other issues the COV feels are
relevant.**

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

The COV sees a need to stress the big-picture perspective by PIs and
reward this perspective appropriately. Also, there is a need to develop
a role for Critical Infrastructure Protection in this program as well as
in other NSF programs.

Given the startup status of the program, this program has been very
successful in starting to build an NSF basic research community in
trusted computing.

The COV recommends a greater effort devoted to infusing security
awareness into other NSF programs.

The COV recommends greater exploration of the multi- and
inter-disciplinary aspects of trusted computing.

For NSF programs to make a difference in areas related to cyber trust,
the traditional NSF basic research community needs to be brought into
sustained contact with more applied and operational expertise. The COV
believes that there is much valuable experience and knowledge in the
corporate and non-profit research and development communities. There
should be some creative thinking within NSF to develop mechanisms and
vehicles for harnessing those resources. Possibilities include
subcontracting, co-PI-ship, teaming and possibly joint projects with
DARPA and other funding agencies.

SIGNATURE BLOCK:

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ 16 June 2003

For the Trusted Computing COV

Peter Neumann and George Cybenko

Co-Chairs

+---------+------------------------------------------------------------+
| Section | ##                                                         |
+---------+------------------------------------------------------------+
| 3       |                                                            |
+---------+------------------------------------------------------------+

## 

## 

Appendices

## 

# Appendix 1. Working Agenda

## Thursday, June 5, 2003

**Room 110**

08:30 AM Coffee and Introductions

08:45 AM Welcoming Remarks, Deborah Crawford, Assistant Director, CISE

09:00 AM Welcoming Remarks, Kamal Abdali, Division Director, C-CR

09:10AM Procedural Details, Frank Anger, Deputy Division Director. C-CR

09:20 AM Explanation of Program Statistics, Carmen Whitson, C-CR

09:45 AM COV Remarks and Instructions, Richard DeMillo, Chair, COV

10:00 AM Breakout into 10 subcommittees (TOC, GSG, CSA, DSC, SEL, DA,
COM, SPS, TC, EHS)

Review jackets and grant results. (Room numbers listed below)

12:00 PM Lunch

01:00 PM Breakout (continued)

04:30 PM Plenary Session to report subcommittee findings and any
problems, to plan for Day 2, and to give out any assignments for the
evening.

05:30 PM Adjourn

06:30 PM Dinner (optional: group reservations will be made)

**Friday, June 6, 2003**

**Room 110**

08:30 AM Coffee

08:45 AM Subcommittee Instructions, Richard DeMillo, Chair, COV

09:00 AM Breakout Sessions

11:00 AM Plenary Session, Subcommittee reports, Discussion of
conclusions with NSF staff, and NSF staff response

12:00 PM Lunch

01:00 PM Breakout sessions, prepare COV report draft

05:00 PM Adjourn

**Breakout Rooms for COV Subcommittee Meetings**

130 Theory of Computing

II-517 Graphics, and Symbolic & Geometric Computation

330 Computer Systems Architecture

365 Distributed Systems and Compilers

370 Software Engineering and Languages

340 Design Automation for Micro and Nano Systems

380 Communications Research

1150 Signal Processing Systems

220 Trusted Computing

517 Embedded and Hybrid Systems

# Appendix 2 COV Roster June 5-6, 2003

## COV Chair

Dean Richard DeMil

College of Computing

Georgia Institute of Technology

<rad@cc.gatech.edu>

### **Software Engineering & Languages**

Professor Victor R. Basili

University of Maryland

basili@cs.umd.edu

Dean Debra J. Richardson

School of Information and Computer Science

University of California, Irvine

mtwyvill@uci.edu

Dr. Tony Wasserman

Software Methods and Tools

<tonyw@acm.org>

### **Graphics, and Symbolic & Geometric Computation**

Professor George Labahn

University of Waterloo

glabahn@scg.math.uwaterloo.ca

Professor Ahmed Sameh

Director, Computing Research Institute

Purdue University

sameh@cs.purdue.edu

Professor Jessica K. Hodgins

Carnegie Mellon University

jkh@cs.cmu.edu

Professor Chandrajit Bajaj

University of Texas, Austin

<bajaj@cs,utexas.edu>

### **Design Automation for Micro & Nano Systems**

Professor Joseph R. Cavallaro

Rice University

cavallar@ece.rice.edu

Professor Robert W. Newcomb

University of Maryland

newcomb@eng.umd.edu

Professor Yannis P. Tsividis

Columbia University

<tsividis@ee.columbia.edu>

**Computer Systems Architecture**

Professor Jean-Luc Gaudiot

University of California, Irvine

gaudiot@uci.edu

Professor Pen-Chung Yew

University of Minnesota

yew@umn.edu

Dr. Kemal Ebcioglu

IBM Watson Research Labs

<kemal@watson.ibm.com>

### **Trusted Computing**

Professor George Cybenko

Dartmouth College

gvc@dartmouth.edu

Dr. Peter G. Neumann

SRI International Computer Science Lab

neumann@csl.sri.com :e-mail

### Embedded & Hybrid Systems

Professor Kane Kim

University of California, Irvine

khkim@uci.edu

Dr. Douglas C. Schmidt

Vanderbilt University

<dschmidt@snap.org>

### **Theory of Computing**

Professor Juris Hartmanis

Cornell University

jh@cs.cornell.edu

Professor Vaughan Pratt

Stanford University

pratt@cs.stanford.edu

Professor Robert Sedgewick

Princeton University

<rs@cs.princeton.edu>

### 

### 

### 

### **Distributed Systems & Compilers**

Dr. Marion Harmon

Florida A&M University

harmon@cis.famu.edu

Professor Kishore Ramachandran

Georgia Institute of Technology

rama@cc.gatech.edu

Hank Levy

University of Washington

<levy@cs.washington.edu>

### **Signal Processing Systems** 

Professor Mark Smith

Professor John Treichler

Cornell University

treichler@cnf.cornell.edu

**Ron Schafer**

Georgia Institute of Technology

<Ron.Schafer@ece.gatech.edu>

### **Communications Research**

Professor Arlene A. Cole-Rhodes

Morgan State University

acrhodes@eng.morgan.edu

Ender Ayanoglu

University of California, Irvine

ayanoglu@uci.edu

Chris Heegard

Native Intelligence

<heegard@nativei.com>

Purdue University

<mjts@purdue.edu>

# Appendix 3 COV Guidelines

SUBCHAPTER 300-COMMITTEE OF VISITORS REVIEWS

**310** **INTRODUCTION**

**320** **COV REVIEW CYCLES,** **SCHEDULES,** **AND TIMING**

**330 COV RELATIONSHIP TO ADVISORY COMMITTEE AND FACA**

**340** **ESTABLISHING** **AND SUPPORTING** **A COV**

350 COV MEMBERSHIP

**360** **SCOPE OF COV** **REVIEW**

370 INFORMATION ACCESSIBLE TO COVS

**380** **COV** **REPORTS** **AND RESPONSES TO COV REPORTS**

**310** **INTRODUCTION**

[311.]{.mark} [COV reviews]{.mark} [provide NSF with]{.mark}
[external]{.mark} [expert judgments]{.mark} [in two areas:]{.mark} [\
(1)]{.mark} [assessments of the quality and integrity of program
operations and program-level technical and managerial matters pertaining
to proposal decisions; and]{.mark} [\
(2)]{.mark} [the degree to which the outputs and outcomes generated by
awardees have contributed to the attainment of NSF's mission, strategic
goals, and annual performance goals.]{.mark}

**320** **COV REVIEW CYCLES,** **SCHEDULES, AND TIMING**

321\. Programs and offices that recommend or award grants, cooperative
agreements, and/or contracts whose main focus is the conduct or support
of research and education in science and engineering are to be reviewed
by a COV at regular intervals of no more than three years.

325\. Programs that are related administratively or programmatically and
can be grouped together within a cluster, section, or division, should
be scheduled for COV review at the same time. Assistant Directors and
O/D Office Directors will select the level of aggregation at which COVs
will be conducted according to the specific needs of the division,
directorate, or office, and to provide the appropriate coverage of
program management.

**330** **COV RELATIONSHIP TO ADVISORY COMMITTEE** **AND FACA**

331\. The Federal Advisory Committee Act (FACA) of 1972 requires that
committees providing advice directly to the management of Federal
agencies be chartered and operated in conformance with the procedural
requirements of the FACA (see NSF Manual 1, Chapter IV, Subchapter
100-Management of Advisory Committees and Review Panels).

332\. Subcommittees, such as NSF's COVs, that report to the chartered
"parent" Advisory Committee, rather than the agency, do not meet the
definition of an advisory committee in the FACA, and thus are not
subject to the procedural requirements of the Act.

333\. NSF COV meetings need not be announced in the Federal Register,
nor are they required to be open to the public.

[334. The COV chairperson, or designee,]{.mark} [reports]{.mark} [the
COV]{.mark} [findings and recommendations to]{.mark} [the affiliated
chartered parent]{.mark} [Advisory]{.mark} [Committee.]{.mark}

[335.]{.mark} [The]{.mark} [affiliated,]{.mark} [chartered
parent]{.mark} [Advisory Committee must review and discuss]{.mark} [each
report from an unchartered COV. The Advisory Committee]{.mark}
[discussion]{.mark} [of the COV report]{.mark} [may take the form of a
face-to-face meeting or]{.mark} [video teleconference]{.mark} [under
regular FACA procedures.]{.mark}

340 ESTABLISHING AND SUPPORTING A COV

343\. The make-up of the COV, including the COV membership and the COV
chair, is the responsibility of the Directorate or Office leadership.

344\. Directorates and Offices will make logistical arrangements,
provide background materials and agency guidelines for the COV, and will
facilitate the efficiency and effectiveness of the COV review.

345\. [Directorates and Offices will appoint a person]{.mark} [who will
have]{.mark} [responsibility]{.mark} for approving the COV agenda, for
orienting the COV members and reviewing the COV responsibilities with
the COV members (including confidentiality, conflict-of-interest, and
Core Questions), and for [ensuring]{.mark} [that]{.mark} [NSF Form
1230P, Conflict-of-Interests and]{.mark} [Confidentiality]{.mark}
[Statement, is filed by]{.mark} [each]{.mark} [COV member]{.mark} [and
reviewed]{.mark} [before the COV meeting, and]{.mark} [appropriate
records are kept.]{.mark}

**350** **COV** **MEMBERSHIP**

351\. The selection of credible, independent experts who are able to
provide [balanced and impartial assessments of NSF programs]{.mark} and
activities is critical to the credibility of the COV mechanism.
Assistant Directors and O/D Office Directors are responsible for the
selection of COV members. They should consult with the appropriate
Division Director in guiding the selection of COV members, and also
consult with the Chair of the affiliated Advisory Committee when
selecting COV members.

352\. Assistant Directors and Office Directors are responsible for
ensuring that the selection of COV members contributes to a [balanced,
independent]{.mark} [review that reflects a diversity of perspectives
and is free from]{.mark} [disqualifying]{.mark} [conflicts of
interest.]{.mark}

353\. To the fullest extent possible, the composition of the committee
should be balanced with respect to programmatic coverage, types of
institutions under review, and geographic representation.

354\. [At least twenty-five percent of the COV]{.mark} [membership
should include qualified individuals who are not currently serving on
any NSF Advisory Committee and have not been applicants to the program
under review for at least five years. Any]{.mark} [person with an action
pending in a program under review]{.mark} [may not]{.mark}
[participate]{.mark} [as a COV member for that program.]{.mark} [COV
membership should also include at least one member of the affiliated
Advisory Committee.]{.mark}

355\. [COV membership should include representatives of those
disciplines, fields and activities that are impacted or affected by the
outputs and outcomes resulting from NSF awards recommended by the
program(s)]{.mark} [under review]{.mark}. Examples include but are not
limited to: separate but related disciplines, private industry,
government agencies and laboratories, educators, foreign scientists and
engineers, and other potential users.

**[360]{.mark}** [**SCOPE OF COV REVIEW**]{.mark} []{.mark}

[]{.mark}

[361.]{.mark} [Each COV review should provide NSF with results and
other]{.mark} [information that can be]{.mark} [integrated]{.mark} [at
the Foundation level, as specified in NSF's GPRA Strategic Plan]{.mark}
[and annual]{.mark} [Performance Plan.]{.mark} [The cognizant Assistant
Directors and Office Directors may also use the COV mechanism to gather
evaluation information on other aspects of program management and
organizational performance and on]{.mark} [issues such as investment
strategies or priority setting of importance to the divisions and
directorates.]{.mark}

[]{.mark}

[362. The COV review of program management will consider proposal
actions that were completed during the previous three fiscal
years.]{.mark}

[]{.mark}

[363. The COV review of awardee results will consider examples of the
direct accomplishments of projects supported by the programs under
review that are either currently active at the time of the COV review or
were closed out during the previous three fiscal years.]{.mark}

[]{.mark}

[364.]{.mark} [A set of]{.mark} [Core Questions and]{.mark} [a]{.mark}
[Reporting Template]{.mark} [for COV use,]{.mark} [reflecting noteworthy
changes in the annual]{.mark} [NSF]{.mark} [Performance Plan, will be
issued and revised periodically by the Office of the Director. The
revised set of questions will be issued in the form of an O/D Staff
Memorandum and will be maintained by OIA.]{.mark}
[(<http://www.insidensf.gov/od/gpra/>).]{.mark} [OIA will provide
assistance to all]{.mark} [Directorates]{.mark} [and]{.mark}
[Offices]{.mark} [using these questions with COVs.]{.mark}

[365.]{.mark} [For each COV review, the cognizant Assistant Director or
Office Director will, in collaboration with the Chair of the Advisory
Committee or equivalent affiliated organization, prepare a written
charge to the COV consisting of the Core Questions as well as other
items of a general and program-specific nature of interest to the
NSF.]{.mark}

[]{.mark}

[366. The COV Core Questions and Reporting Template will be applied to
the program portfolio and will address the]{.mark} [proposal]{.mark}
[review]{.mark} [process used by the program, program
management,]{.mark} [and the results of NSF investments. Specific
questions to be addressed and reported on are:]{.mark}

[]{.mark}

[a) the integrity and efficiency of processes used to solicit, review,
recommend]{.mark} [and document]{.mark} [proposal actions, including
such factors as:]{.mark}

[selection of an adequate number of highly qualified reviewers who are
free from bias and/or conflicts of interest;]{.mark}

[appropriate use of NSF merit review criteria;]{.mark}

[documentation related to program officer decisions regarding awards and
declines, and the scope, duration and size of projects;]{.mark}

[balance of awards in terms of subject matter;]{.mark} [emerging
opportunities; high risk and innovation;]{.mark} [size versus number of
awards;]{.mark} [new investigators; diversity of underrepresented
groups; geographic distribution of principal investigators; and]{.mark}

[overall technical management of the program.]{.mark}

[]{.mark}

[b) the relationships between award decisions, program goals, and
Foundation-wide programs and goals;]{.mark}

[]{.mark}

[c) results,]{.mark} [in the forms of outputs and outcomes of NSF
investments for the relevant fiscal years, as they relate to the]{.mark}
[Foundation's]{.mark} [current]{.mark} [strategic]{.mark} [goals]{.mark}
[and]{.mark} [annual performance goals.]{.mark}

[]{.mark}

[d) the significant impacts and advances that have developed since the
previous COV review and are demonstrably linked to NSF investments,
regardless of when these investments were made. Examples might include
new products or processes, or new fields of research whose creation can
be traced to the outputs and outcomes of NSF-supported projects over an
extended period of time.]{.mark}

[]{.mark}

[e) response of the program(s) under review to recommendations of the
previous COV review (see 378).]{.mark} []{.mark}

[]{.mark}

[367. NSF staff may assist a COV technically in the preparation or
review of a COV report only to the extent of ensuring that the
confidentiality of NSF material is not compromised and to correct errors
of a factual nature.]{.mark}

370 INFORMATION ACCESSIBLE TO COVS

371\. COV members will be informed [that the names and comments of
reviewers are confidential, and that information related to declinations
is confidential.]{.mark}

372\. [COV members will not review any proposal jacket with which they
have]{.mark} [been determined to have]{.mark} [a
conflict-of-interest.]{.mark} [COV members will not review any proposal
from their own institution or any proposal]{.mark} [in which they
collaborated or participated.]{.mark}

373\. [COVs will have access to:]{.mark} []{.mark}

[documentation of completed proposal actions]{.mark} [for the period
under review;]{.mark}

[a list of all projects closed out during the period under
review;]{.mark}

[annual program reports covering the period under review which
include]{.mark} [summary data on the portfolio]{.mark} [relevant
to]{.mark} [activities in]{.mark} [the program]{.mark} [(per Section 366
and 375);]{.mark}

[examples of results in the form of outputs and outcomes from current
and recently completed projects;]{.mark}

[-]{.mark} [examples of connections between outputs and outcomes from
NSF projects and significant impacts]{.mark} [or]{.mark} [advances
resulting]{.mark} [from NSF-supported projects;]{.mark} [and]{.mark}

[-]{.mark} [the]{.mark} [most recent]{.mark} [COV report.]{.mark}
[]{.mark}

374\. [COV members will have access to all program documentation
completed during the period under review, including proposals. The
COV]{.mark} [members will review a balanced sample of proposals
representative of the portfolio of the program, including awards,
declines, borderline proposals, proposals returned as inappropriate, and
proposals that were withdrawn. The]{.mark} [COV members]{.mark} [may
choose to review proposals]{.mark} [through a random]{.mark} [selection
process of their own.]{.mark}

375\. To facilitate the COV review, [each program will prepare an annual
summary report on program performance that covers the period under
review]{.mark} and reflects the portfolio of activities for the program,
including Foundation-wide activities. Centralized NSF data modules
should be used where possible. Examples of the information and data that
should be provided in the annual report include:

Total number of proposals received in the program;

Numbers of proposals awarded, declined, withdrawn, and returned as
inappropriate;

Data and funding rates for categories of PIs;

Methods of merit review used;

Data pertaining to categories of reviewers (including new PIs and
members of underrepresented groups);

Information on issuance of new solicitations and guidelines;

Participation and role of the program in cross-cutting and NSF-wide
activities;

Number of new projects, continuing projects, and projects closed out;

Examples of awardee results (with award number) reported during the
period under review including:

outputs and outcomes resulting from projects in progress or closed out
during the period under review;

significant impacts and advances linked to the outputs and outcomes of
NSF investments made at other times; and

results relevant to NSF's mission, and to outcome goals and areas of
emphasis that apply to the period under review.

376[.]{.mark} [NSF program officers will provide appropriate]{.mark}
[examples of awardee]{.mark} [results for each year under review]{.mark}
[in the annual program report]{.mark} (see Sections 363 and 375).
Results may be selected from the Annual and Final Project Reports of NSF
funded projects that were on-going or closed out during the period under
review, and examples of results from other sources. The information
provided might vary depending on the type of project supported and how
it relates to the goals of the activity being reported and/or NSF-wide
goals. Examples selected should reflect the most significant
accomplishments in a program's portfolio of support within the reporting
period, and may identify topics or areas requiring special consideration
or attention by the program. It is not necessary to select samples from
each project closed out during the previous three fiscal years: COV
members will be provided with a list of all projects closed out during
the period under review, and will have access to final project reports
upon request.

377\. Results from NSF supported research and education projects may
lead to discoveries and achievements in areas beyond the immediate
project and may eventually have broader impacts such as service to
society. The connections between the initial project and broader impacts
may be anecdotal. In preparing annual program reports throughout the
year, program staff should look for examples of links and connections
between results of awards and NSF goals, and strive to present examples
of such connections in annual program and performance reports and for
COV reviews whenever possible.

378\. The COV members will be provided with the most recent COV report
and the related unit's response to that report.

**[380]{.mark}** []{.mark} [**COV**]{.mark} **[REPORTS AND]{.mark}**
**[RESPONSES TO COV REPORTS]{.mark}** []{.mark}

[]{.mark}

[381.]{.mark} [The]{.mark} [COV chairperson]{.mark} [will furnish a
written report to]{.mark} [the parent Advisory Committee]{.mark} [or its
delegated representative.]{.mark} [The parent Advisory
Committee,]{.mark} [or delegated representative,]{.mark} [will transmit
the COV report(s)]{.mark} [to NSF management and request NSF]{.mark}
[management]{.mark} [to respond to recommendations made]{.mark}
[in]{.mark} [the COV]{.mark} [report(s).]{.mark} []{.mark}

[]{.mark}

[The COV report:]{.mark}

[will]{.mark} [address the COV charge, describe the method of review,
including the number of files it examined and method for selecting them,
and the nature of other information provided by the programs under
review, and will include any COV findings, recommendations, or
suggestions;]{.mark}

[b.]{.mark} []{.mark} [will]{.mark} [include all components of a
standard COV Reporting Template]{.mark} [as provided in the Core
Questions, which are]{.mark} [approved by the NSF Director and
maintained by OIA]{.mark}
[([http://www.insidensf.gov/od/gpra/](http://www.insidensf.gov/of/gpra/));]{.mark}

[c.]{.mark} [may also]{.mark} [include additional sections]{.mark}
[specific]{.mark} [to the programs under review.]{.mark}

[]{.mark}

[]{.mark} [383. The]{.mark} [COV]{.mark} [report will be transmitted by
the COV]{.mark} [chairperson]{.mark} [to the appropriate]{.mark}
[Advisory Committee]{.mark} [(or delegated representative)]{.mark}
[within two weeks of the COV meeting, and at least four weeks prior to
the Advisory Committee meeting at which the report will be
reviewed.]{.mark}

384\. When received, the Advisory Committee chairperson, or designated
representative, will transmit the COV report to the cognizant Assistant
Director or Office Director and request a written response.

385\. The Assistant Director or Office Director will

a\. prepare a written response to the COV report. The response should
describe action(s) to be taken on each suggestion or recommendation
where action is appropriate.

b\. prepare a document describing the diversity, independence, and
balance represented by the members of the COV, and the resolution of
real or apparent conflicts of interest.

c\. transmit the written response and the document describing diversity
and conflict resolution to the Advisory Committee and NSF senior
management.

d\. [return any COV report to the Advisory Committee, or delegated
representative, which does not adequately]{.mark} [address the Core
Questions.]{.mark}

386\. The following COV documents will be made available by the
Assistant Director or Office Director in electronic format within 15
days of the Advisory Committee review of the COV report. Notification of
access to documents should be provided to the NSF Director, Chief
Operating, Financial and Information Officers, the NSF Inspector
General, Office of Integrative Activities Director, and the NSF
Committee Management Officer:

the COV report, including a list of COV members and charge to the COV;

the Assistant Director's or Office Director's report on COV membership
and resolution of conflicts as specified in Section 385.b;

the Assistant Director\'s or Office Director's response to the COV
report(s);

the parent Advisory Committee\'s approval (written comments if
appropriate)relevant to the COV report(s), if any; and a list of
Advisory Committee members.and

387\. The Assistant Director or Office Director [will also make
available to the public, upon request, printed copies of the
documents]{.mark} specified in Section 386.

# Appendix 4 COV Template

##### CORE QUESTIONS and REPORT TEMPLATE

#####  for 

##### FY 2003 NSF COMMITTEE OF VISITOR (COV) REVIEWS

**Guidance to the COV:** The COV report should provide a balanced
assessment of NSF's performance in two primary areas: (A) the integrity
and efficiency of the ***processes*** related to proposal review; and
(B) the quality of the ***results*** of NSF's investments in the form of
outputs and outcomes that appear over time. The COV also explores the
relationships between award decisions and program/NSF-wide goals in
order to determine the likelihood that the portfolio will lead to the
desired results in the future. Discussions leading to answers for Part A
of the Core Questions will require study of confidential material such
as declined proposals and reviewer comments. *COV reports should not
contain confidential material or specific information about declined
proposals**.*** Discussions leading to answers for Part B of the Core
Questions will involve study of non-confidential material such as
results of NSF-funded projects. It is important to recognize that the
reports generated by COVs are used in assessing agency progress in order
to meet government-wide performance reporting requirements, and are made
available to the public. Since material from COV reports is used in NSF
performance reports, the COV report may be subject to an audit.

*We encourage COV members to provide comments to NSF on how to improve
in all areas, as well as suggestions for the COV process, format, and
questions.*

##### FY 2003 REPORT TEMPLATE FOR

#####  NSF COMMITTEES OF VISITORS (COVs)

+-----------------------------------------------------------------------+
| ##### Date of COV                                                     |
+-----------------------------------------------------------------------+
| **Program/Cluster:**                                                  |
+-----------------------------------------------------------------------+
| **Division:**                                                         |
+-----------------------------------------------------------------------+
| **Directorate:**                                                      |
+-----------------------------------------------------------------------+
| **Number of actions reviewed by COV**[^64]**: Awards: Declinations:   |
| Other:**                                                              |
+-----------------------------------------------------------------------+
| **Total number of actions within Program/Cluster/Division during      |
| period being reviewed by COV**[^65]**: Awards: Declinations: Other:** |
+-----------------------------------------------------------------------+
| **Manner in which reviewed actions were selected:**                   |
+-----------------------------------------------------------------------+

**PART A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures. **Provide comments in the space below the
question**. **Discuss areas of concern in the space provided.**

+-----------------------------------------------------------+----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES**  | **YES,   |
|                                                           | NO,**    |
|                                                           |          |
|                                                           | **DATA   |
|                                                           | NOT      |
|                                                           | AV       |
|                                                           | AILABLE, |
|                                                           | or NOT   |
|                                                           | APPL     |
|                                                           | ICABLE** |
+-----------------------------------------------------------+----------+
| Is the review mechanism appropriate? (panels, ad hoc      |          |
| reviews, site visits)                                     |          |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Is the review process efficient and effective?            |          |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Are reviews consistent with priorities and criteria       |          |
| stated in the program's solicitations, announcements, and |          |
| guidelines?                                               |          |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Do the individual reviews (either mail or panel) provide  |          |
| sufficient information for the principal investigator(s)  |          |
| to understand the basis for the reviewer's                |          |
| recommendation?                                           |          |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Do the panel summaries provide sufficient information for |          |
| the principal investigator(s) to understand the basis for |          |
| the panel recommendation?                                 |          |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Is the documentation for recommendations complete, and    |          |
| does the program officer provide sufficient information   |          |
| and justification for her/his recommendation?             |          |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Is the time to decision appropriate?                      |          |
|                                                           |          |
| Comments:                                                 |          |
+-----------------------------------------------------------+----------+
| Discuss issues identified by the COV concerning the       |          |
| quality and effectiveness of the program's use of merit   |          |
| review procedures:                                        |          |
+-----------------------------------------------------------+----------+

A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers. **Provide comments in the space below the question.
Discuss issues or concerns in the space provided.**

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       |           |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments:                                                |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     |           |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments:                                                |           |
+----------------------------------------------------------+-----------+
| Have the *review analyses* (Form 7s) addressed whether   |           |
| the proposal contributes to both merit review criteria?  |           |
|                                                          |           |
| Comments:                                                |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system.               |           |
+----------------------------------------------------------+-----------+

A.3 Questions concerning the selection of reviewers. **Provide comments
in the space below the question. Discuss areas of concern in the space
provided.**

+---------------------------------------------------------+------------+
| Selection of Reviewers                                  | YES , NO,  |
|                                                         |            |
|                                                         | DATA NOT   |
|                                                         | AVAILABLE, |
|                                                         |            |
|                                                         | or NOT     |
|                                                         | APPLICABLE |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       |            |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            |            |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    |            |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      |            |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments:                                               |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
+---------------------------------------------------------+------------+

A.4 Questions concerning the resulting portfolio of awards under review.
Provide comments in the space below the question. Discuss areas of
concern in the space provided.

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **            |
|                                                      | APPRORIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     |               |
| projects supported by the program.                   |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  |               |
| scope of the projects?                               |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| -   High Risk Proposals?                             |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| -   Multidisciplinary Proposals?                     |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| -   Innovative Proposals?                            |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| -   Funding for centers, groups and awards to        |               |
|     individuals?                                     |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| -   Awards to new investigators?                     |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| -   Geographical distribution of Principal           |               |
|     Investigators?                                   |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| -   Institutional types?                             |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance of:                                          |               |
|                                                      |               |
| -   Projects that integrate research and education?  |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       |               |
| balance:                                             |               |
|                                                      |               |
| -   Across disciplines and subdisciplines of the     |               |
|     activity and of emerging opportunities?          |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          |               |
| participation of underrepresented groups?            |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      |               |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments:                                            |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
+------------------------------------------------------+---------------+

A.5 Management of the program under review**. Please comment on:**

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments:                                                             |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments:                                                             |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments:                                                             |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
+-----------------------------------------------------------------------+

##### PART B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

**NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.**

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institutions.

+-----------------------------------------------------------------------+
| **B.1 [NSF OUTCOME GOAL for PEOPLE]{.underline}: Developing "a        |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| Comments:                                                             |
+-----------------------------------------------------------------------+
| **B.2 [NSF OUTCOME GOAL for IDEAS]{.underline}: Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| Comments:                                                             |
+-----------------------------------------------------------------------+
| **B.3 [OUTCOME GOAL for TOOLS:]{.underline} Providing "broadly        |
| accessible, state-of-the-art and shared research and education        |
| tools."**                                                             |
|                                                                       |
| Comments:                                                             |
+-----------------------------------------------------------------------+

##### PART C. OTHER TOPICS

C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.

C.2 Please provide comments as appropriate on the program's performance
in meeting program-specific goals and objectives that are not covered by
the above questions.

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

C.4 Please provide comments on any other issues the COV feels are
relevant.

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the \[Replace with Name of COV\]

\[Name of Chair of COV\]

Chair

# Appendix 5. Chair's Presentation

[^1]: CCR-9972532

[^2]: CCR-9902022, CCR-9988165, and CCR-0092532

[^3]: To be provided by NSF staff.

[^4]: To be provided by NSF staff.

[^5]: To be provided by NSF staff.

[^6]: To be provided by NSF staff.

[^7]: CCR-9910084

[^8]: CCR-0084252, DMS-0072540, DMS-0073056, DMS-0070641, DMS-0071412,
    DMS-0084438, DMS-9973303, DMS-9973341

[^9]: CCR-9624315

[^10]: CCR-9624315

[^11]: CCR-9732220

[^12]: CCR-9981754

[^13]: CCR-9701508

[^14]: CCR-9902032

[^15]: CCR-9988535

[^16]: CCR-9902009

[^17]: CCR-9900333

[^18]: CCR-9988177

[^19]: CCR-9988357

[^20]: CCR-9972532

[^21]: CCR-9902022, CCR-9988165, and CCR-0092532

[^22]: To be provided by NSF staff.

[^23]: To be provided by NSF staff.

[^24]: To be provided by NSF staff.

[^25]: To be provided by NSF staff.

[^26]: To be provided by NSF staff.

[^27]: To be provided by NSF staff.

[^28]: To be provided by NSF staff.

[^29]: To be provided by NSF staff.

[^30]: To be provided by NSF staff.

[^31]: To be provided by NSF staff.

[^32]: To be provided by NSF staff.

[^33]: To be provided by NSF staff.

[^34]: To be provided by NSF staff.

[^35]: To be provided by NSF staff.

[^36]: NCO SDP Coordinating Group Report on New Visions in Software
    Design and Productivity, Vanderbilt University, Nashville, TN Dec.
    14-15, 2002.

[^37]: Source: Hans Mark speech to the National Defense Industrial
    Association Science and Technology Conference, held from May 9-11,
    2000

[^38]: Source: ITO Study on Next-generation Information Technology
    Challenges, 1999.

[^39]: Helen Gill, Shankar Sastry, John Stankovic, Janos Sztipanovits:
    Report of the NSF/OSTP Workshop on Innovative Information
    Technologies for Critical Infrastructure Protection, National
    Conference Center, Sept 19-20, 2002.

[^40]: K. H. Hodges, Embedded Micro-component Market Study, DARPA.

[^41]: R. H. Bourgonjon, "Embedded Systems in Consumer Products," in
    *Lecture Notes on Embedded Systems*, LNCS Vol. 1494, 1996, pp.
    395-403.

[^42]: Gunter Heiner, DaimlerChrysler, Proc. of the Joint Workshop on
    Advanced Real-Time Systems, Vienna, 26 March 2001.

[^43]: Tom Fuhrman, GM, The Role of Embedded Software in the Automotive
    Industry, ARIES Workshop, CMU, April 19, 2002.

[^44]: President of the United States report on the "Status of Federal
    Critical Infrastructure Protection Activitiesm" January 2001, p. 79.

[^45]: Joint NSF/DARPA Planning Workshop on Embedded and Hybrid Systems,
    Arlington, VA, May 2001.

[^46]: Hybrid Systems Computations and Control (HSCC), Palo Alto, June
    2001.

[^47]: Embedded Software (EmSoft) Workshop, Lake Tahoe, NV, October
    2001.

[^48]: NSF/ACM Workshop on Embedded Computing (SIGBED), Atlanta, GA,
    November 2001.

[^49]: NCO SDP Coordinating Group Report on New Visions in Software
    Design and Productivity, Vanderbilt University, Nashville, TN Dec.
    14-15, 2002.

[^50]: Helen Gill, Shankar Sastry, John Stankovic, Janos Sztipanovits:
    Report of the NSF/OSTP Workshop on Innovative Information
    Technologies for Critical Infrastructure Protection, National
    Conference Center, Sept 19-20, 2002.

[^51]: IEEE Computer Society\'s Int\'l Symp. on Object-oriented
    Real-time distributed Computing (ISORC), Crystal City, VA, April
    2002 (Special Session on National-Level R&D Movements in Software
    Technologies) .

[^52]: IEEE Computer Society\'s Workshop on Object-oriented Real-time
    Dependable Systems (WORDS), Guadalajara, Mexico, Jan. 2003.

[^53]: James Keller, Joel Esposito, Vijay Kumar, George Pappas: Embedded
    and Hybrid Systems, NEPHEST Report, July 17, 2002.

[^54]: Special issue on Model-Based Design of Embedded Software January
    2003.

[^55]: Special Issue on Real-Time Systems, 2003.

[^56]: IEEE Transaction on Embedded Systems.

[^57]: IEEE/ACM Embedded Software (EMSOFT) Conference.

[^58]: National Research Council: Embedded, Everywhere, 2001.

[^59]: NCO HCSS Coordinating Group Report on High Confidence Software
    and Systems, 2001

[^60]: NCO SDP Coordinating Group Report on New Visions in Software
    Design and Productivity, 2002.

[^61]: Workshop on Embedded Computing, Princeton University, November
    15-26, 2001.

[^62]: To be provided by NSF staff.

[^63]: To be provided by NSF staff.

[^64]: To be provided by NSF staff.

[^65]: To be provided by NSF staff.