based upon only a part of a more comprehensive program of experiments.
It is difficult therefore to judge what should or should not be done
since the work described was very incomplete in itself.

Technology Transfer/Application

This set of presentations, along with two presented as a part of
Boundary Layer/Turbulence Research, seemed to be a representative
selection of work by the Environmental Applications Branch (EAB). That
is, the presentations were consistent the function of the EAB.

The panel regards the UNAMAP to be of very good quality and to be a
most important function of the MD. Indeed, UNAMAP is one of the most
important and visible aspects of the EPA’s air quality operations,
since regulatory applications (e.g., obtaining permits, establishing
emission limits, or demonstrating attainment of air quality standards)
typically require the use of a UNAMAP model.

The estimation of the reduction of soybean crops resulting from
various levels of ozone was presented. This analysis was requested by
the Office of Air Quality Planning and Standards. Current levels of
ozone at 320 farms growing soybeans were related to existing and
proposed national ambient air quality standards (NAAQS) for ozone. An
important finding from this research was that, contrary to greenhouse
studies, field studies of the effect of ozone on soybean growth
suggested that two parameters of soybean exposure to ozone (namely, the
duration and the concentration of the exposure) could be summarized by
a single index (effective mean ozone concentration). This work seems
to the panel to be highly responsive to the needs of EPA policy
analysis, and like previous work at the MD on the relationship between
concentration averaging times, may be widely used and cited. Due to
the importance of this work, it deserves further investigation and the
collaboration of other experts in meteorology, atmospheric chemistry,
plant physiology, and statistical inference.

The panel rated as very important the investigation of alternative
techniques for randomly sampling data to efficiently estimate long-term
(seasonal or annual) concentrations using the Point, Area, and

16


APPENDIX F
ASRL STAFF RESPONSE TO REVIEWERS’ COMMENTS


APPENDIX E
RESULTS OF THE EPA PARTICIPANT SURVEY
METEOROLOGY DIVISION


Response: We agree. A journal article describing the in-house effort on devel-
opment of a biogenic hydrocarbon emissions inventory is in progress, although
transmittal of results to the forest and crop researchers has been ongoing. We
also agree with the reviewers that the uncertainty estimates are questionable.
The intention of the presentation was to provide uncertainty estimates for bio-
genic emissions using the same methodology currently used by the National Acid
Precipitation Assessment Program (NAPAP) for the anthropogenic emissions inven-
tory. Thus, at least the relative uncertainties of the biogenic and anthropo-
genic emissions could be compared. NAPAP recognizes the weakness of the current
methodology but funding for improvements in this area is limited.

Comment, p. 10: Work has focused on ozone dry deposition over large areas as
determined from aircraft measurements... An effort should be made to complete
the analysis of these unique data sets by interpretation of convective struc-
tures (for -all fields) and treating ozone as a tracer. This has some exciting
possibilities.

Response: In-house analysis of the aircraft turbulence ozone measurements has
indeed been focused on the dry deposition of ozone as noted by the reviewers

The major purpose of the turbulence aircraft flight program during the Northeast
Regional Oxidant Study (NEROS) was to obtain experimental data about the spatial
variation and temporal behavior of ozone dry deposition over representative land
use types. The in-house research to date was intentionally designed to study the
variability of ozone fluxes and deposition velocities for different land use
areas as functions of time/height in support of the existing in-house regional
oxidant model program. There are certainly other interesting aspects of this
data set, such as venting of ozone by convective cloud elements and vertical
ozone transport over urban areas; these topics have been examined through an
extramural effort that was not part of this peer review. Other reviewer-
suggested enhancements to the program would be interesting and productive to
accomplish, but resources are limited and we must focus on program needs.

Comment, p. 11: Feedback between the physicist(s) behind the model development
and programming quality control people was a little vague. Scientists that
develop the model know the inherent weaknesses. Quality control should gener-
ate, independently, a similar set of weaknesses.

: Interaction between the model developer, Bob Lamb, and the quality
control staff is significant even though this aspect was not stressed in the
presentation. Any suspect data is immediately brought to the attention of Dr.
Lamb who further directs an assessment of the cause and effect of the anomalous
data. Summary graphics such as contours of the maximum daily ozone and the QC
tracer species are provided to both Dr. Lamb and to Ken Schere, the model evalu-
ator, on a routine basis.

: What about vectorizing the code for the ROM?... Although
briefly discussed, the ROM may be improved more by a parallel implementation
than by a vector implementation.

Response: About one year ago, IBM personnel made a serious attempt to vectorize
the ROM code. Benchmark executions showed no improvement in performance of the
vectorized code. Subsequent studies performed by the Research Triangle Insti-
tute (RTI) explained why ROM is not effectively vectorized, but more likely to

42


derivation of relevant biomass volumes, and identification of species
emission factors. This approach appears appropriate for the intended
use and necessarily involves extrapolation and disaggregation of
existing information. The research has involved contact with forest
and crop researchers during the development of information sources, but
has not resulted in presentation of results to that community. The
latter is recommended as a complement to existing reports to the air
pollution modeling community. Uncertainty estimates were presented,
but seemed questionable.

Work has focused on ozone dry deposition over large areas as
determined from aircraft measurements. Data were collected through
extramural effort, but data analysis has occurred in-house. Aircraft
Measurements suggest that ozone concentrations may be highly correlated
to organized convection. Ozone appears to be a good tracer of
convection, similar to moisture. Detrended temperature should be
spectrally decomposed to show the basic convective mode (BCM) or cloud
street mode. An approximate 4 km length scale is evident in the raw
data. The ozone spectrum presented indicates a 3.8 km spectral peak,
which accords well with temperature data. What about the spectrum for
the specific humidity q? Also, several papers exist in the literature
that show larger length scales of mesoscale convective organization in
a convective boundary layer, known as higher convective modes (HCM).
The typical aspect ratio for the BCM is 3 to 1, compared to an order of
magnitude increase for the larger HCMs. Inspection of the raw
concentration for ozone indicates a signal on the larger length scale
that is in phase with temperature trace. An effort should be made to
complete the analysis of these unique data sets by interpretation of
convective structures (for all fields) and treating ozone as a tracer.
This has some exciting possibilities.

The development and implementation of verification procedures for
the ROM will significantly enhance the current and future use of ROM.
Current computer science concepts have been integrated into EPA’s
atmospheric modeling community. This integration not only results in
products of known quality, but also appears to be accomplished as a
natural part of the model development. There appears to be an active

10


°.
SZ 2 UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
F ATMOSPHERIC SCIENCES RESEARCH LABORATORY
ae pont RESEARCH TRIANGLE PARK
NORTH CAROLINA 27711

DATE: February 11, 1988

SUBJECT: Release of Peer Review Panel Report: In-House
Research Program By The Meteorology Division

FROM: Ronald K. Patterson, Peer Review Coordinator LoL)
Atmospheric Sciences Research Laboratory (MD-59)

TO: H. Matthew Bills, Acting Director
Office of Acid Deposition, Environmental Monitoring
and Quality Assurance (RD-680)

THRU: Alfred H. Ellison, Director
Atmospheric Sciences Research Laboratory (MD-59)

The enclosed peer review panel report has been reviewed and released
by our Laboratory. Comments by ASRL participants and our Laboratory
Director are included in the report under Appendices F and G, respectively.
We find the comments from our participating staff to be complete and appro-
priate.

This peer review evaluated only the in-house research program under
the ASRL Meteorology Division. A total of 45 technical presentations
were made during the four day peer review. The enclosed peer review
panel report is highly complimentary of the Division's leadership and the
competence of the technical staff. The review panel was impressed by the
high degree of excellence exemplified in the areas of experimental fluid
dynamics and computational quality control. However, the review panel
expressed concern regarding inadequate computer resources and the external
pressures exerted on the staff to prematurely release scientific findings.
The review panel also discussed the need to strengthen in-house statisti-
cal expertise.

We are distributing copies of this report and cover letter to the
individuals listed below. The ASRL peer review contractor, Research and
Evaluation Associates, Inc., has been instructed to forward a- copy of
this final released version of this report to each peer review panelist.

Enclosure

cc: Erich Bretthauer (RD-672) Courtney Riordan (RD-682)
William Keith (RD-680) Deran Pashayan (RD-680)
Morris Altschuler (RD-674) Thert Wiser (ANR-443)
Elenora Karicher (RD-680) Jack Shreffler (MD-59)
Basil Dimitriades (MD-59) William Wilson (MD-59)

Frank Schiermeier (MD-80) Meteorology Division Staff (MD-80)


APPENDIX G
THE LABORATORY DIRECTOR’S REVIEW COMMENTS
ON THE PANEL REPORT AND THE ASRL STAFF RESPONSE


APPENDIX B
AGENDA FOR EPA-ASRL PEER REVIEW OF
METEOROLOGY AND ASSESSMENT DIVISION
SEPTEMBER 15-18, 1987


sD 5%
s "es.

ct)

UNITED STATES ENVIRONMENTAL PROTECTION AGENCY

ATMOSPHERIC SCIENCES RESEARCH LABORATORY
1 eRe RESEARCH TRIANGLE PARK
NORTH CAROLINA 27711

atotltng
2
OW agenct

as

MEMORANDUM
DATE: January 15, 1988

SUBJECT: Response to Peer Review Comments on Meteorology and
Assessment Division In-House Research Programs

FROM: Francis A. Schiermeier, Director
Meteorology and Assessment Division, ASRL mh
TO: Ronald K. Patterson

ASRL Peer Review Coordinator

The peer reviewers' report on the Meteorology and Assessment Division (MD)
in-house research programs is both comprehensive and enlightening. We appreciate
their significant efforts, considering that forty-five technical presentations
were directed at the reviewers during the three-day schedule.

The report contains comments and recommendations on program strengths and
weaknesses, on the adequacy of personnel staffing and computer facilities, on
maintaining the critical balance between research and applications, and on the
Division's role in establishing research priorities. The responses provided
here are addressed to critiques, recommendations, and reviewer questions in all
of these areas. Responses are not provided for the complimentary statements
contained in the report although they have been noted with appreciation.

3 ve Summar Repeated recommendations on the
need to strengthen in-house statistical expertise to support model evaluations.

We agree. We are currently recruiting a Ph.D. statistician for the
Division but our efforts are hampered by a temporary NOAA hiring freeze.

Comment, p. 4: The Meteorology Division seems to be a frequent recipient of
requests from other offices and laboratories within the EPA that require redi-
rection of program FTE's. The Division appears to be responding to these
requests, but at the cost of delaying longer term research needed by the Agency.
In some instances...scientific expertise is steered away from model development
to application runs. Some of these "quick turnaround" demands for staff time
seem to be appropriate uses of expertise within the MD; however, the balance of
short-term and long-term activities should be defined more explicitly.

: We agree with the ideal of a clearly defined balance between research
and applications in the Division but we must also realize that the EPA is a reg-
ulatory Agency. As such, it is subject to frequent crises-oriented mandates for
redirection of research efforts imposed by Congress and the Administration.

38


SECTION 1

INTRODUCTION

This is a report from the Peer Review Panel (see Appendix A for
list of panel members) organized to review the in-house research
activities of the Meteorology Division (MD) of the US Environmental
Protection Agency (EPA), Atmospheric Sciences Research Laboratory
(ASRL). Prior to meeting with the MD staff, the panel received three
volumes of selected research papers by the MD staff for preliminary
review. On September 15-18, 1987, the panel met at the Research
Triangle Park, NC, to hear 45 presentations by the MD staff on in-house
activities. Presentations focused on individual scientific
contributions to EPA programs rather than a review of activity as
project officers or monitors of extramural contracts.

The panel appreciates the efforts of the MD Director and individual
staff members in preparing well-organized preview material and
informative presentations. In general, the quality and organization
were equivalent to that of a scientific professional meeting. In
organizing this report, the panel chose to fotlow the general
scientific areas that were used to organize the presentations (see
meeting agenda in Appendix B).

Finally, the panel is grateful to Charlotte Coley (Research and
Evaluation Associates) and Ron Patterson (ASRL Peer Review Coordinator)
for assistance with the logistics in conducting the review and
preparing this report.


TABLE 1 SUMMARY OF PROCESS EVALUATIONS

Number of Reviewers
Rating Each Item
Review Categories Very
Poor Fair Good Good Excellent
Preview Materials
1. Written Quality 1 4
2. Technical Quality 2 3
3. Utility for Outside Reviewer 1 1 3
4. Adequacy of Time Available
To Preview 1 1 2 1
SUB TOTALS 1 2 6 uI
Process and Logistical Information
5. Meeting Purpose 1 2 1 1
6. Scheduling of Meeting 1 1 3
7. Reviewer Responsibilities:
Time/Preparation Requirement 3 1 1
8. Overall Peer Review Process 1 1 3
9. Timeliness of Meeting
Notification 1 4
10. Timeliness of Logistical
Information 1 4
SUB TOTALS 2 9 6 13
Review Meeting
11. Adequacy of Time for
Discussion w/EPA Staff 3 2
12. Adequacy of Time for
Executive Session 4 1
13. Quality and Utility of
Presentations 1 1 3
14. Quality and Utility of
Materials Disseminated 1 4
15. Support Services and
Activities 1 1 3
SUB_ TOTALS 10 3 12
TOTALS 3_| 21 | 15 36

31


Line-Source (PAL) algorithm. This work addresses a practical need for
computational efficiency, but, more significantly, explores
probabilistic modes of air quality analysis. This work should continue
and should be supported by a statistician who is an expert in the
techniques and inferential issues of data sampling schemes.

Finally, a preliminary diagnosis of the Turbulent Profile Sigmas
(TUPOS) dispersion model based on a comparison of calculated and
measured SFg concentrations obtained from the EPRI’s Kincaid data base
was presented. The investigation will help to determine the degree to
which more detailed turbulence data improves model performance under
field conditions and is therefore regarded as extremely important by
the panel. As with other model diagnoses, sensitivity studies, and
reliability studies conducted by the Division, the statistical issues
are quite challenging (see Section 3 and Appendix C of this review) and
deserve the support of an expert statistician.

17


10:30 - 11:00 AM
11:00 - 11:30 AM

11:30 - 12:30
12:30 - 01:00 PM
01:00 - 01:30 PM

01:30 - 01:40 PM

01:40 - 02:00 TMM
02:00 ~- 02:20 PM

02:20 - 02:40 PM
02:40 - 03:00 PM

03:00 - 03:20 PM

03:20 - 03:35 PM

03:35 - 03:50 PM

03:50 - 04:10 PM

04:10 - 04:30 M

04:30 - 05:00 PM

Regional Acid Deposition Model
(RADM) Evaluation Research Program

Robin Dennis

Cumulus Venting of Pollutants
from the Mixed Layer

Jason Ching

Lanch
Executive Session for Review Panel

Acid Deposition Assessment
Studies

Robin Dennis

Complex Terrain Dispersion Modi
Overview of Complex Terrain

Research

Frank Schiermeier

Complex Terrain Field Data Bases Larry Truppi

Fluid Modeling Facility Cbject- Bill Snyder
ives, Facilities, Approach, and
Recent Research Accomplishments

Deformation of Plumes by Hills Bill Snyder

Break
Measurements of Streamline Traj-

ectories in Stratified Flow Over
Isolated Hills

Roger Thompson

Laboratory Measurements of
Unsteadiness of Strongly Strat-
ified Flow Fields rag) Over
Two-Dimensional Hills

Bill Snyder

Comparison of Numerical and
Laboratory Experiments on Density-
Stratified Flow Over a Hill

Roger Thompson

Stratified Flow Over Ridges and Bob Lawson
Valleys (Flushing of Valleys)

Laboratory Validation of Flat-
Dividing-Streamline-Surface
Approximation for Complex Terrain
Dispersion Models

Bill Snyder

Executive Session for Review
Panel and MD Division Director

23

Frank Schiermeier

20

23

24
25

26

28

29

30


EPA-ASRL Peer Review Panel
Meteorology Division
Research Triangle Park, North Carolina
September 15-18, 1987

Ernest M. Agee

Chairman, Dept. of Earth and
Atmospheric Sciences

Purdue University

West Lafayette, IN 47907

(317) 494-3282

Raymond J. Deland

Raymond Deland & Associates
2 Collyer Drive

Ossining, NY 10562

(914) 941-5622

David Emmitt

Simpson Weather Associates, Inc.
809 E. Jefferson Street
Charlottesville, VA 22902

(804) 979-3571

P.K. Misra

Ministry of the Environment
125 Resources Road

East Wing

Rexdale, Ontario

Canada MSW SL1

(416) 235-5771

Anthony R. Olsen

Battelle Pacific Northwest Program
P.O, Box 999

Richland, WA 99352

(509) 376-4265

Anthony D. Thrall

Environment Risk Assesment Program
Electric Power Research Institute
3412 Hillville Aveme

Palo Alto, CA 94304

(415) 855-2627

19


Comments:

"The organization and support of the peer review by R & E, the
Meteorology Division director and his staff is by far the best
of all prior ASRL reviews that I have participated in. They
are to be commended for their efforts. Reviews of in-house
research by a Division are new and require further
clarification. The review has elements of personnel action
review/appraisal, university tenure review, departmental
accreditation, and "traditional" EPA program reviews. It is
difficult for peer reviewers to define their task within the
existing framework."

"Impossible to examine all preview materials."
"Entire process is too brutal.”
"Too many people to review with diverse activities."

"The purpose of the review must be carefully considered. My
impression is that the review was foisted on ASRL by
Washington, DC. To their considerable credit, ASRL management
shared with us this perspective on the review. But I feel
that the use to the ASRL of such reviews can be better defined
and expressed to future panels, after some reflection by the
ASRL management."

Recommendations:

"It would have been nice to have some one-on-one discussion
with selected staff."

"Up-to-date vita should be provided."
"EPA MD organizational chart would have been useful."
"A time to speak with key people on a one-to-one basis."

"Provide organizational chart to all panel members in advance
of meeting. Also, as part of putting the research being
reviewed in perspective, it would have been useful to have had
a few pages describing the Meteorology Division’s mission,
long-term objectives, and recent accomplishments - a
description at a level of detail somewhere between reference
item #1 and individual workplans."

32


The cloud venting research focuses on the study of penetrative
convection, which leads to venting of the planetary boundary layer and
transport of pollutants. The results of this work have important
implications for both the ROM and RADM programs and should be
continued. The researcher is encouraged to establish stronger research
relationships with the cloud dynamics community. The field experiment
data are unique and deserve more detailed documentation of the
meteorology surrounding the events. The further classification and
determination of 8 factors for several types of cloud-topped
(fractionally) planetary boundary layer is recommended.

Overall, the MD staff involved in regional acid deposition model ing
are (1) performing research that is very relevant to EPA programs, (2)
actively interacting with the relevant scientific community, and (3)
completing research that is worthy of publication (and is being
published) in the peer-reviewed literature.

Complex Terrain Dispersion Modeling

Nearly 30% of the presentations were associated with in-house
efforts to support the Complex Terrain Dispersion Modeling (CTDM)
program. The panel was unanimous in its positive evaluation of the
quality of the experimental modeling activity. The ability and
willingness of the fluid modeling team to respond to the demands of a
regulatory agency and still be able to carry out basic research speaks
well for both the scientists and division management.

Since the panel was not briefed on the complete CTDM program, it
was unable to review the in-house research within the context of the
whole project. However, the following specific comments can be made:

1) Good, interesting results are obtained in the fluid modeling
laboratory and should continue to be published in JFM type journals.
The panel, however, questions whether the laboratory experiments are
sufficiently reliable simulations of atmospheric flows to be used to
set regulatory guidelines.

2) Inverted tows in stratified cases are adequate, however a deep
sheared flow layer is not present (as in the neutral wind tunnel case

14


SECTION 4
PROGRAM ASSESSMENT

To facilitate the achievement of consensus, the committee adopted an
internal evaluation procedure whereby each panel member assigned the
value of 5 (Excellent), 4 (Very Good), 3 (Good), 2 (Fair), or 1 (Poor)
for each of the following performance criteria:

e Quality of Science/Work

. Relevance to MD/EPA Missions

e Interaction with In-House Colleagues and Outside
Community

Emphasis was given to scientific accomplishment, such as reviewed
journal publications and conference/workshop publications and
presentations. Professional service to EPA and extramural activities
were also considered. In some instances, scientific support was the
focus of work activity rather than basic research or model development
and testing. Recognition was also given to the importance of applied
research. Although the scores are not provided in this report, the
panel directed itself to write program evaluations that were consistent
with composite numerical scores assigned to the performance criteria.
The panel was pleased with the consistency of the individual
evaluations. The reviews given below reflect more directly on some
presentations than others. Some individual comments are provided, but
not in detail for each individual presenter. Absence of detailed
technical comments should not be construed as positive or negative.


APPENDIX D

ASRL PROCESS EVALUATION FOR METEOROLOGY DIVISION
IN-HOUSE RESEARCH PEER REVIEW


APPENDIX C
COMMENTS ON MODEL EVALUATION


SECTION 3
GENERAL OVERVIEW

The MD programs are conducted in a favorable in-house research
support environment. Generally, the scientific staff seems to be
enthusiastically engaged in their efforts, and morale within the
Division seems good. Administrative leadership is effective in
promoting this favorable professional environment. MD programs are an
important part of both national and international research efforts, and
the reputation of the staff brings respect and requests for applying
in-house scientific expertise to help address the needs of EPA’s policy
analysis.

The experimental fluid dynamics is excellent and goes far in
advancing the MD program objectives. The computational quality control
of model development is exemplary and in itself represents an ideal
model for QC assessment that could be transferred outside the Division.

However, computer resources are inadequate for model development,
applications, and evaluation. Computer cycles seem to be highly
variable and inadequate, and thus introduce some degree of uncertainty
in carrying out the MD programs. Small computers are augmenting
computing needs, but a total spectrum of computing support is required
(from small work stations that stand alone or access other machines to
supercomputer model simulations for numerically intensive model
development and testing).

In-house statistical expertise should be strengthened to support
model evaluations. The activities known variously as "model
evaluation" and "model validation" deserve more attention by
environmental researchers, in general, and thus by the MD in its role
as a leader of environmental research. There seem to be essentially
two purposes of these activities: (1) to better understand atmospheric
processes, and thus to improve atmospheric models, by identifying and
diagnosing failures in model performance, and (2) to quantify the
reliability of models that are used either to decide regulatory issues


SECTION 2
CHARGE TO REVIEW PANEL

The panel was provided with the Division Director’s preview
materials outlining the mission of the Division’s program. Principles
to guide the peer review process were stated in the Research and
Evaluation Associates, Inc. document Guide for Participants in the EPA-
ASRL_Task/Project Peer Review Process provided by Charlotte Coley.
Further discussions with the MD Director at the beginning of the site
visit helped to clarify the charge to the review panel. Evaluation of
in-house research and support activities given in this report are based
on the following program objectives:

e Basic Research

e Model Development

e Applications and Evaluations

The panel has strived to evaluate the research presented without
the benefit of some important information such as CVs that describe and
quantify total up-to-date professional activities. Although the panel
was provided with detailed individual workplans, insufficient time was
available to ingest all this information.

The panel attempted to evaluate the in-house activity, realizing
that a sizeable portion of the comprehensive program effort resides
within the extramural community. Further, the panel has taken into
consideration that MD program activities (although residing within the
ORD) must respond to regulatory needs.


studies). Experimental fluid mechanics is done well, both in
laboratory techniques (flow visualization and measurement) and
application of theory. How these results are factored into the CTDM
was not explained well.

3) Relevance of model and laboratory results (even with good
agreement) do not necessarily apply to atmospheric boundary layer
flows. Stably stratified flows are useful, but do not address the
convective planetary boundary layers. Dynamic similarity between
numerical model/laboratory flow and atmospheric analog was not
established. It appears that a constant eddy viscosity value
arbitrarily replaces the molecular viscosity in the model momentum
equation. In reality, isotropic Fickian diffusion cannot be assumed.
Further, the spatial dependencies in nonlinear systems of equations
give rise to non-Gaussian dispersion. Again, the fluid mechanics work
is excellent but its extension to actual atmospheric dispersion is not
clearly explained.

Wake Effects Research

Although driven by regulatory requirements, the wake effects
research can be considered quite basic. The work on the
characterization of the complex turbulence in the wake of a
non-aerodynamic structure is a good example. The need for an improved
plume formulation is obvious and common to much of the theoretical and
physical modeling efforts at MD. Efforts to obtain better flow
visualization and, in particular, to quantify those observations are to
be commended. Illumination techniques and the statistical evaluation
of the resulting data need to be improved. New plume formulations call
for a departure from the traditional Gaussian approach. Any
fundamental advances in this area would have an impact not only on EPA
activities but also on the boundary layer turbulence research community
in general.

The work on auto exhaust dispersion is clearly associated with
EPA’s need to model the impact of a major source of atmospheric
pollution. The experimental results presented during the review were

15


for establishing the MD’s research priorities. A more active role by
the MD in setting research priorities would help to guard against the
erosion of the MD’s research capability by an overload of quick
response tasks.

Finally, the emerging program concept of Advanced Research Model
Implementation is strongly endorsed by the panel. Several existing
modeling efforts form a nucleus for this initiative. An additional
consideration should be given to nesting some of these models inside

the global climate models that presently exist within the research
community.


AGENDA
Peer Review of the EPA-ASKL

Meteorology and Assessment Division

In-House Research Program
September 15-18, 1987

08:00 — 08:20 AM

08:20 — 08:40 AM

08:40 - 09:00 AM

09:00 — 09:10 AM
09:10 — 09:40 AM

09:40 — 10:00 AM
10:00 - 10:30 AM
10:30 - 11:00 AM
11:00 - 11:30 AM
11:30 - 12:30 AM

12:30 = 01:00 PM
01:00 - 01:20 PM

‘TOPIC

Classroom 3, Environmental

Research Center Research Triangle

Park, North Carolina

Opening Session - Panel Welcome Charlotte Coley
P Review P ori :
ASRL Peer Review Coordinator Ron Patterson

Executive Session in Meeting Al Ellison

Room for Panel and Staff, ASRL
Director

Coffee Break and Introductions of
Review Panel and ASRL Staff

Introduction by MD Director
Boundary Layer/Turbulence Research

Frank Schiermeier

Analysis and Parameterization Gary Briggs
of Convective Diffusion

Processes

Evaluation of Convective Scaling Tom Pierce

for Estimating Diffusion

Turbulence Studies Jason Ching
in Urban and Non-Urban Areas
Rotary Spectral Analysis of Peter Finkelstein
Turbulence Measurements

Exponential-Sum-Fitting Tech-
niques for On-Site Turbulence
Analysis

Tanch

Steve Perry

Executive Session for Review Panel

Meteorological Scaling in Applied John Irwin
Dispersion Modeling

21


SED SI,
»* "es,

a

gnnoulany,

Atmospheric Sciences Research Laboratory

3
yy CHEMISTRY, MATHEMATICS, METEOROLOGY, MODELING, PHYSICS
Pay prove

PEER REVIEW REPORT
ON THE
IN-HOUSE RESEARCH PROGRAM BY THE
METEOROLOGY DIVISION

September 15-18, 1987

Panel
Ernest M. Agee, Chair
Raymond J. Deland
David Emmitt
P.K. Misra
Anthony R. Olsen
Anthony D. Thrall

Office of Acid Deposition, Environmenta! Monitoring and Quality Assurance
Office of Research and Development
U.S. Environmental Protection Agency
Research Triangle Park, North Carolina 27711


Recent correspondence with the NWS National Environmental Satellite Data and
Information Service concerning the GOES NEXT generation of satellites shows
Promise that, in the coming decade, sufficiently complete data sets will be
archived to allow subsequent estimations of vertical cloud definition to be made
by our air quality models.

Comment, p. 12: The MESOPUFF II modeling effort is probably important in prin-
ciple, but (very good) performance evaluation shows poor performance. The CAP-
TEX data set appears adequate to test performance. Best results were obtained
for cases of no directional shear. Work should be published in reviewed jour-
nals (not just an EPA report).

Response: The in-house effort to evaluate and test the MESOPUFF II regional
episodic model with the CAPTEX data base consisted of: (1) an operational evalu-
ation in which the model was applied with all default features; (2) diagnostic
model runs to explore differences with alternative wind fields and dispersion
features in the model; and (3) model sensitivity test runs which primarily
focused on variations in SOx from exercising model options and changing key
parameters in the dry deposition and chemistry modules. The presentation at the
Peer review showed just a sampling of some of the many statistical evaluation
results and graphical displays of observed and model tracer plume paterns. The
analysis and interpretation of model results were aimed at identifying reasons
for model overprediction and spatial displacements between the respective plume
positions. While complete results of this task are being documented in an EPA
report, there are plans to publish the notable results in other publications as
recommended.

: In addition to the statistical evaluation, additional effort is
needed to outline conditions under which the models are or are not applicable.
For example, the simple Lagrangian models show the same performance repeatedly
over several years when annual averages are compared. The ISDME analysis would
be more useful if it had led to an understanding of why the models show this
type of performance.

Response: The goals of the International Sulfur Deposition Model Evaluation
(ISDME) focused on the performance of eleven long-term models for each season of
one year. These evaluation periods were commensurate with the periods for which
the models were designed to simulate the processes. However, to identify the
conditions under which these models are or are not applicable, one first must
investigate the behavior of the models for periods much shorter than a_ season,
i.e., for a week or less. Although this was once an objective of the ISDME,
participating modelers did not submit results for periods considerably less than
their interpretation of the appropriate modeling period. Nonetheless, an in-
house investigation is currently underway to characterize the behavior of one of
these models for daily periods within one season of the ISDME evaluation year.
From these results, we should be able to determine those meteorological situa-
tions for which the model is or is not applicable.

: The cloud venting research focuses on the study of penetrative
convection, which leads to venting of the planetary boundary layer and transport
of pollutants. The results of this work have important implications for both
the ROM and RADM programs and should be continued.

44


the quality of the work is good. Such work may not enhance our
understanding of boundary layer turbulence significantly. The
interaction with the outside community is not very apparent.

Similarly, the analyses of turbulence data using superposition of
autocorrelation functions is not very relevant. The quality of the
work is fair and interaction with outside communities not very good.
Eulerian autocorrelation functions do not determine diffusion in a
turbulent flow. Relating Eulerian autocorrelation functions to
Lagrangian autocorrelation functions is not simple and unique. Also,
"true" Eulerian autocorrelation functions in the atmosphere exhibit
oscillation about the zero line thus making the estimate of time scales
questionable.

Regional Oxidant Modeling

This effort within the Division epitomizes the full spectrum of
program objectives ranging from modeling development to applications
and evaluation. The nucleus of staff spearheading this effort is
highly competent and is attempting to entrain appropriate activities
throughout the division. It seems that external pressures (outside of
ASRL/MD) are responsible in part for this program initiative. These
pressures also require model results that are difficult to achieve.
Imposed resolutions in the Regional Oxidant Model (ROM), as required to
be practical, are inconsistent with routine meteorological data sets.
For example, upper air weather observations are taken every 12 hours on
a mean spatial scale of 325 km. Model integrations are for one hour
over an 18x18 km grid. The ROM appears to be a zero-order model in
atmospheric dynamics (at best), whereas the chemistry of the 28
reactive species is treated in a more sophisticated manner. Further,
the raw input data of emissions are rather discretized and irregularly
available in space and time. The ROM is a pure diagnostic model, i.e.
output concentrations are used (after the fact) to evaluate the effects
of alternative strategies for managing air quality planning and
Standards emission. It is not realistic to impose point concentrations
(observations and/or standards) on regional model plume dispersions.

8


undar er/Turbi ice_Res:

The ASRL is actively involved in the following areas of boundary
layer and turbulence research:

1. Experimental investigation of atmospheric boundary layer
turbulence,

2. Innovative dispersion model development,

3. Evaluation and assessment of dispersion models.

The committee evaluated these areas of research in terms of their
quality, relevance to EPA goals, and interaction of the responsible
scientists with the outside scientific communities.

Experimental boundary layer research includes turbulent diffusion
experiments inside a convective boundary layer and urban and non-urban
boundary layers. The quality of research in these areas is excellent.
This research is very relevant to EPA objectives and the responsible
scientists have excellent interactions with outside communities.

New modeling approaches are being investigated using probability
density functions of vertical velocity in convective boundary layers.
For buoyant plumes, semi-empirical models have been developed. Again,
the quality of research in these areas is excellent, the work is highly
relevant, and there is very good interaction with outside communities.
Future work in this area should emphasize better characterization of
buoyant plume dispersion inside a convective boundary layer since the
probability density function approach is not expected to work well. The
quality of work in the application of models is very good, and the work
is highly relevant and interactive with the outside community. It is
good to see the efforts directed towards definition of atmospheric
stability with proper boundary layer velocity and length scales. The
sensitivity analyses of the models could be improved by using a joint
probability density function of the variables rather than varying them
independently.

The rotary spectral density application in the analyses of
turbulence data is not obviously relevant to EPA objectives even though

7


Comment, p. 8: Similarly, the analyses of turbulence data using superposition

of autocorrelation functions is not very relevant. The quality of the work is
fair and interaction with outside communities not very good. Eulerian autocor—
relation functions do not determine diffusion in a turbulent flow. Relating
Eulerian autocorrelation functions to Lagrangian autocorrelation functions is
not simple and unique. Also "true" Eulerian autocorrelation functions in the
atmosphere exhibit oscillation about ‘the zero line thus making the estimate of
time scales questionable.

Response: Concerning the relationship between the Eulerian and Lagrangian
autocorrelation functions, the interpretation of Eulerian autocorrelation func-
tions as an indication of the turbulent eddy structure is certainly not new;
Sutton (1953) mentions this topic. Although the Eulerian autocorrelation func-
tion does not directly determine turbulent diffusion, the literature is replete
with attempts to better estimate and interpret the form of this function; some
examples are Misra (1979) and several cited by Moore et al. (1985). Much has
been published (for example, Hanna 1981, 1986; Weber et al. 1982) showing the
usefulness of estimates of Lagrangian statistics (such as integral time scales)
from Eulerian statistics for applications in diffusion modeling. Regarding the
contact with the outside community, the autocorrelation study was presented (and
received favorable comment) at the AMS Fifth Joint Conference on Applications of
Air Pollution Meteorology (Dahai 1986).

=9: The Regional Oxidant Model (ROM) appears to be a zero-order
model in atmospheric dynamics (at best)... The ROM is a pure diagnostic model,
i.e., output concentrations are used (after the fact) to evaluate the effects of
alternative strategies for managing air quality planning and standards emission.
ROM development apparently needs more computing cycles. Algorithms for the model
are not fully understood (for example, incorporation of the effect of clouds and
radiative processes).

Response: While the reviewers acknowledge the purely diagnostic use of the ROM
for assessing emissions control strategies on a regional basis, there seems to
be some confusion on the nature of the model itself. There are inferences to
the use of better predictive techniques for the atmospheric dynamics portion of
the model. The wind fields used in the ROM are derived from observational data
and physical principles using a new technique described in Lamb and Hati (1987)
that permits explicit treatment of the uncertainty in describing atmospheric
motion. This method permits full utilization of dynamical principles in con-
structing flow fields for the ROM, but only a portion of the capabilities of the
technique have been implemented to date. In addition, flow in the night-time
radiation inversion is simulated using a submodel based on solutions of the
shallow water equations.

Therefore, if the reference to a "zero-order" model pertains to the flow field
description used in the ROM, it is incorrect. The wind field specifications are
at least "first-order" accurate in their satisfaction of dynamical constraints.
In addition, the ROM incorporated schemes for describing vertical mass flux,
including cumulus cloud fluxes, horizontal transport, deposition, subgrid scale
chemistry effects, etc., that were designed specifically to describe regional
scale transport and chemistry phenomena. Unfortunately, there was not enough
time in the peer review to describe the numerous facets of the ROM model in any

detail.
40


analysis of model sensitivities may require the statistical design of
simultaneous perturbations of several variables.

Uncertainty analysis is the estimated attribution of the
discrepancies between measurements and calculations to, respectively,
errors in model input, errors in ambient measurements, model bias, and
random variation of the atmosphere for each set of conditions defined
by the model input variables. In statistical parlance this is called
variance components analysis (if characterization of conditional
variability is restricted to conditional means and variances).
Uncertainty analysis is relevant to model diagnosis. For example, it
can be used to determine whether the apparent poor performance of the
model under certain conditions can be plausibly attributed to
Measurement errors or random atmospheric variation rather than model
bias. Uncertainty analysis also pertains to model reliability. That
is, the amount of evidence showing that the model "is" or “is not"
biased under certain conditions, determines the degree to which such
findings generalize beyond the study to other settings or sites.
Uncertainty analysis is extremely challenging since the atmosphere
rarely provides more than one replicate of the same set of model input
conditions. However, the use of either statistical data resampling
schemes or wind-tunnel data as a test-bed may yield useful information.

In all four types of "model evaluation", it is essential that the

evaluation team include or have access to appropriate statistical
expertise.

28


Response: We agree. However, NAPAP budget cuts not only prevented further
field evaluation studies but even prematurely terminated analysis of field
measurement data that had been previously collected.

Comment, p. 14: Good, interesting results are obtained in the fluid modeling
laboratory and should continue to be published in JFM type journals. The panel,
however, questions whether the laboratory experiments are sufficiently reliable
simulations of atmospheric flows to be used to set regulatory guidelines.

Response: We believe the proper answer to the reviewers’ question is, "not if
taken in isolation." The results of laboratory experiments have not been used,
in and of themselves, to set regulatory guidelines. Many of the studies at the
Fluid Modeling Facility (FMF) have been done upon request from the Office of Air
Quality Planning and Standards (OAQPS). It is within the purview of the OAQPS
to utilize the laboratory results along with many other inputs in setting such
guidelines. As a prime example, the dividing-streamline concept arose from FMF
laboratory experiments. This concept was validated through the complex terrain
field program, then implemented in the Complex Terrain Dispersion Model (CTDM)
It is now up to the OAQPS to establish how the results of the CTDM program,
including the results of the laboratory experiments, are to be used in the regu-
latory process.

Comment, pp. 14-15: Inverted tows in stratified cases are adequate, however a
deep sheared flow layer is not present (as in the neutral wind tunnel case
studies). Experimental fluid mechanics is done well, both in laboratory tech-
niques (flow visualization and measurement) and application of theory. How
these results are factored into the CTDM was not explained well.

Response: The primary value of the work in the stratified towing tank is in
providing fundamental understanding of the physical processes involved in stra-
tified flows. Much can be learned concerning the structure of stratified flows
even in the absence of velocity shear layers. Many aspects of these flows are
dominated by the density stratification rather than the velocity shear. Earlier
work (not reported to the review panel) showed that under very strongly strati-
fied conditions, the shear layer is unimportant. The incorporation of labora-
tory results (and complementary field results) into the CTDM was done by the
contractor and therefore was not discussed in detail at this in-house review.

Comment, p. 15: Stably stratified flows are useful, but do not address the
convective planetary boundary layer.

Response: The addition of a convective tank at the Fluid Modeling Facility is
presently in progress. The most important question to be answered concerning
stratified flows in complex terrain is "where is the plume", e.g., impinging on
the windward slope, going over the top, or being directed around the side. The
question of diffusion about the plume centerline is generally of secondary
importance.

: Although driven by regulatory requirements, the wake effects
research (non-aerodynamic structures) can be considered quite basic... Efforts
to obtain better flow visualization and, in particular, to quantify those obser-
vations are to be commended. Illumination techniques and the statistical evalu-
ation of the resulting data need to be improved.

45


COMMENTS ON MODEL EVALUATION

It seems useful to distinguish the following four types of model
evaluation:

(1) model diagnosis

(2) reliability estimation

(3) sensitivity analysis

(4) uncertainty analysis

Model diagnosis is the comparison of measurements to model
calculations so that inadequate assumptions or formulations embodied in
the model can be identified and investigated. The investigation often
leads to model improvements, so that model diagnosis is typically one
phase of model development. Such diagnoses are often informal data
analyses used to generate new hypotheses. Of course, if such an
informal data analysis suggests a major redirection of research or
model development, the diagnosis is worth checking more rigorously.

Estimating the reliability of a model requires more careful and
thorough statistical analyses than does model diagnosis. First, the
use or uses of the model for which the reliability analysis is being
conducted must be carefully defined. Then a means of estimating the
model’s reliability must be determined. Finally, the degree to which
the results of analysis generalize to other conditions must be judged.
These issues are just beginning to be addressed by environmental
researchers and require the full attention of both statisticians and
atmospheric scientists.

Two modes of analysis intermediate to model diagnosis and
reliability estimation are sensitivity analysis and uncertainty
analysis. The goal of sensitivity analysis is to determine the
influence of model input variables or model coefficients on some aspect
of the model output. Like model diagnosis, the purpose of sensitivity
analysis is to better understand the model, at least under some
conditions. Such probing of the model is sometimes performed very
roughly by perturbing just one variable at a time to approximately
bound the plausible range of ambient concentrations. A more thorough

27


or to guide regulatory policy. (See Comments on Model Evaluation given
in Appendix C.)

The Meteorology Division seems to be a frequent recipient of
requests from other offices and laboratories within the EPA that
require redirection of program FTEs. The Division appears to be
responding to these requests, but at the cost of delaying longer term
research needed by the Agency. In some instances (e.g. the requests to
run the ROM simulations for EPA constituents), scientific expertise is
steered away from model development to application runs. Some of these
"quick turnaround" demands for staff time seem to be appropriate uses
of expertise within the MD; however, the balance of short-term and
long-term activities should be defined more explicitly.

One of the major challenges to ORD Management would seem to be
finding and preserving a balance of staff time allocated to quick
response needs of the Agency on the one hand and longer-term needs on
the other. Although a good balance seems to have been achieved by the
MD, it also seems to the panel that ORD Management in general has a
responsibility to both the scientific staff and to the Agency to take
steps to guard against the erosion of the Agency’s research capability.

A recommendation of the panel with respect to this issue is to
broaden the criteria used to evaluate the achievements of the
scientific staff. In general, the staff members are doing an excellent
job of communicating with their peers within their respective
disciplines through publications and presentations. But communication
across disciplines (horizontal communication) should be emphasized, for
in the relatively young field of environmental research,
interdisciplinary communication is less well established, although
extremely important.

Similarly, more of the staff should be encouraged to familiarize
themselves with the Agency’s use of their research to help set policy
(vertical communication). This will help to ensure the relevance of
individual research within the MD. More importantly, it should
engender a vision and a coherent, internally developed agenda for the
MD’s research. Even though the MD’s tasks are ultimately determined
outside the Division, such an agenda would be an important reference

4


APPENDIX A
PEER REVIEW PANEL


TABLE 2. EPA PARTICIPANT SURVEY SUMMARY

Soe eee nee ee esses eee esse a ER

Number_of Participants Rating Each Item
REVIEW FORMAT & Very Very

LOGISTICS Satisfied|Satisfied|Dissatisfied|Dissatisfied

1. Adequacy of time
for group discussion

with reviewers 1 9 3
2. Adequacy of time for
Project presentations 3 7 3

3. Adequacy of time for
individual discussion
with reviewers 2 7 4

SUBTOTALS 6 23 10

REVIEW PANEL

4. Reviewers’ familiarity
with preview materials 2 10 1

5. Reviewers’ demonstrated
knowledge of your

program 2 10 1
6. Reviewers’ expertise in
the field 4 8 1

7. Selection of reviewers
(Quality of the panel
as_a reviewing unit) 4 8 1

SUBTOTALS 12 36 4

YOUR_ASSESSMENT OF THE

PEER REVIEW PROCESS

8. Support service &
activities carried out

by contractor 3 9
9. Objectivity & profes-

sionalism 3 9
10. Effectiveness as a

mechanism for QA iB 2
SUBTOTALS 6 29 2
TOTALS 24 88 16

35


Comments:

"The schedule was overly crowded; it should have been
stretched to 5 days, with so many presentations."

"Two of the reviewers seémed familiar with air pollution
modeling and meteorology; the others were much less familiar
with this sub-field of meteorology. Although they may be
professional meteorologists, that is no guarantee they are
knowledgable in all aspects of the field.

"This review was unusual in that so many presentations had to
be compressed into three days. However, it still seems that
most presenters had adequate time for interaction with
reviewers as long as they cooperated by allowing time for
questions."

"As is usually the case, there was insufficient time allowed
for the large number of topics presented."

“Very disappointed in not having an opportunity to set up
slides before my presentation because the group did not break
for coffee as scheduled."

"I find that the handouts (copies of visual aids) are
distracting. The audience begins leafing through the material
instead of paying attention to the speaker. I believe the
handouts should be done away with."

"Did not believe the process has much benefit for anyone
except it provides a defense if attacked. In other words, we
can say ‘well, the peer reviewers like our program, why are
you attacking my budget’. Otherwise, the effort is in essence
of no use."

36


Thursday, Sept. 17
08:00 — 08:20 AM

08:20 - 08:50 AM
08:50 — 09:30 AM
09:30 - 09:50 AM
09:50 - 10:20 AM
10:20 - 10:40 AM
10:40 - 11:10 AM

11:10 - 11:30 AM

11:30 - 12:30 TM
12:30 = 01:00 PM

01:00 - 01:30 PM

01:30 - 01:50 PM

01:50 — 02:10 PM

02:10 - 02:30 PM

02:30 - 02:50 PM

Wind Direction Effects on
Dispersion from Sources Downwind
of Hills

Complex Terrain Amplification

Factors for Various Stack Heights/

Locations

Diffusion from Low Level Sources
Under Extreme Stratification

Coffee Break and Introductions of
Review Panel and ASRL Presenters

Wak Res

Theoretical Modeling and Eval-
uation of Building Wake Effects

servations of Surface Flow
Patterns Near Buildings Using
Flow Visualization

Video Image Techniques for Wird
Tunnel Measurements of Building
Wake Dispersion

Near-Wake Dispersion of Auto
Exhaust

Lanch

Executive Session for Review Panel

Technology Transfer /Appl icati

User's Network for Applied
Modeling of Air Pollution
(UNAMAP)

Development and Application of
Air Quality Effects Models

Climatological Version of the
Point Areas, and Line Source
Algorithm (PAL) ,

Bill Snyder
Bob Lawson

Bert Eskridge

Alan Huber

Bob Lawson
Alan Huber

Roger Thompson

Bruce Turner

Ralph Larsen

Bill Petersen

Evaluation of the TUPOS Dispersion Bruce Turner

Model = Preliminary Findings
Break

24

32

33

34

35

36

37

38

39


EXECUTIVE. SUMMARY

The Review Panel has reached a consensus on the following findings
and recommendations:

e Leadership of the Division is highly effective

e Scientific staff is competent

° Scientific morale is good

e Experimental fluid dynamics is excellent

e Computational quality control of model development is
exemplary

e Computer resources are inadequate

e External pressures are forcing premature release of
preliminary scientific findings

e In-house statistical expertise should be strengthened to
support model evaluation

e A program for advanced research model implementation is
strongly endorsed


benefit from a parallel processing implementation. Therefore, a cooperative
research program with RTI is currently implementing a loosely coupled multipro-
cessor architecture for the ROM.

Comment, p. 11: A more coordinated plan is apparently needed for providing
computing facilities to ASRL/MD or even to all of EPA at Research Triangle Park.
Also, how much do QC checks slow down the ROM simulation?

Response: Since the EPA has acquired an additional IBM 3090 E computer, the
computing power of the Agency appears to be sufficient for current and antici-
pated ROM applications. ADP timeshare funding for ROM applications is always a
limiting factor in the number of ROM executions. However, the requirement for
consistent QC checks on the numerous files produced for ROM input is also a
limiting factor in the number of new episode periods to be run. Execution and
QC of three days of model inputs can require up to a week. This processing
involves approximately 60 hours of VAX 785 CPU. Fortunately, the time required
to prepare for a control strategy execution is typically less than a day.

Comment, p.11: Also, can EPA use the supercomputer resources at the five
national centers?

Response: ASRL is also working closely with the EPA National Computer Center to
evaluate the need for supercomputer resources within the EPA. A recent study by
RTI has concluded that the EPA should acquire shared access or its own (mini)
supercomputer at least to explore the potential of the vector supercomputer for
some of its future environmental modeling. However, current OMB funding regula-
tions restrict the use of research dollars for ADP expenditures. Creative solu-
tions may be necessary to use external supercomputer resources.

: The cloud process module work may evolve into good science...
The simplistic approach presented is a reasonable application of the state-of-
the-art knowledge for simple cloud model formulation (primarily a thermodynamic
emphasis). Effort needs to be extended to properly treat the total entrainment
process between the cloud top and the overlying interfacial layer.

: We agree that the current approach to the cloud process module is
simplistic; what was presented was an initial in-house effort. We are currently
exploring the use of a non-hydrostatic three-dimensional cloud model for examin-
ing some of the most important questions such as cloud top and lateral entrain-
ment.

Comment. p..12: The TINEPH data may be alright for radiation studies but its
value in the wet deposition and cloud processing modules is very questionable.
Efforts should be made to incorporate the GOES imagery, which is very useful for
the ROM domain.

Response: We agree. However, the GOES imagery for the 1980 test scenario for
the Regional Particulate Model (RPM) were not available in sufficient complete-
ness to allow determinations of cloud base and top heights to be made, in addi-
tion to the normal areal coverage that most applications require. The RTNEPH
data sets already contain this vertical cloud information, as estimated by the
Air Force through a best possible method approach.

43


: Since a diagnostic model approach is acceptable, more emphasis
should be placed on collecting a more representative data set for model develop-
ment and performance evaluation.

Response: We agree. The regional oxidant program has never been funded suffi-
ciently to mount its own field evaluation study. However, as a last resort, we
plan to join in the field study for the Regional Acid Deposition Model (RADM) to
acquire some appropriate measurement data for evaluation of the ROM.

t .9: The ROM model should interact more with numerical weather pre-
diction model efforts.

Response: Driving the transport portions of air quality models with the output
of numerical weather prediction models is a relatively new concept, now being
used by a few larger scale models such as the RADM and ADOM for the assessment
of acid deposition. This approach is incompatible with the basic premise con-
cerning the stochastic nature of atmospheric motion that underlies the method of
Lamb and Hati used by the ROM. The integration of four-dimensional data assimi-
lation into prognostic wind models is more suited to our needs and it may be
explored in the course of implementing our own wind analysis procedure.

: ...it has been hypothesized that receptor sites can be grouped,
such that concentrations averaged over all sites in a given receptor class at a
given hour is a quantity that is approximately the same for all realizations of
the concentration ensemble. If this is true, then with respect to the receptor
class average, the concentration field is a quasi-deterministic variable. There-
fore, model simulations of single realizations can be interpreted in the conven-
tional deterministic manner. Although this hypothesis has not yet been tested,
it is assumed for the present purposes that it is correct. It is unscientific
to proceed in this way, though it may be politically and practically sound.

Response: The ROM was developed for use in a probabilistic mode. That is, the
model output concentrations for a given receptor site should take the form of an
expected probability distribution of concentrations, reflecting the stochastic
nature of the interpolated flow fields used by the model. The development of
the probabilistic flow model has lagged far behind the development on the other
components of the ROM system. We therefore now have a fully working model except
for this component. Rather than hold up the entire system while awaiting the
completion of the probabilistic flow model (which may yet take considerable
time to complete), we have chosen to proceed in the quasi-deterministic manner
described where observations at aggregates of receptor sites may be matched with
corresponding aggregates of model predictions. While it is true that this
approach necessitates the use of a-priori assumptions on how the receptors may
be grouped, we feel that while imperfect, it is a better route than either put-
ting the mode] on hold while awaiting the final flow model or using the model in
a fully deterministic manner which would be completely inappropriate.

Comment, p. 9-10: Biogenic emissions inventory development is essential for the
application of ROM... The research has involved contact with forest and crop
researchers during the development of information sources, but has not resulted
in presentation of results to that community. The latter is recommended as a
complement to existing reports to the air pollution modeling community. Uncer-
tainty estimates were presented but seemed questionable.

41


effort to publish this approach within the air pollution modeling
community. Although not directly presented during this peer review, it
is known that this experience has contributed to the formulation of
quality assurance procedures for the RADM model evaluation program.

The presence of the Data Systems Analysis Branch staffed with
knowledgeable application computer scientists is one key to the
Meteorology Division’s ability to fulfill their mission. Its presence
allows atmospheric scientists and computer scientists to contribute
their skills to an interdisciplinary effort of model development,
evaluation, and application.

Very systematic and methodical quality control analysis is
performed with state of the art practices, from modular programming
down to file naming conventions. Professional systems programming
architecture could serve as a model for other similar unrelated model
developments. Feedback between the physicist(s) behind the model
development and programming quality control people was a little vague.
Scientists that develop the model know the inherent weaknesses.
Quality control should generate, independently, a similar set of
weaknesses. The need for more computing cycles is not abundantly
clear, but probably is the case. Funds should not be used to upgrade
antiquated computing equipment. What about vectorizing the code for
the ROM? Also, how much do QC checks slow down the ROM simulation? A
more coordinated plan is apparently needed for providing computing
facilities to ASRL/MD or even to all of EPA at Research Triangle Park.
Can metacode be saved, upon which variations of model predictions can
be made? Although briefly discussed, the ROM may be improved more by a
parallel implementation than by a vector implementation. Also, can EPA
use the supercomputer resources at the five national centers?

Regional Particulate Modelin

The Regional Particulate Model (RPM) program uses ROM and has some
of the same problems of weak scientific support and inadequate input
data, especially emissions. The work on different aspects of the
model, presented by four of the scientific staff, showed evidence of

11


activities. Hence no attempt is made to review either past or planned
field experiments or the entire RADM model evaluation effort. Since
the United States portion of the ISDME was completed within the MD, it
will be more fully reviewed. :

The ISDME work has placed the MD staff at the forefront of the
regional model evaluation scientific community. The ISDME Project
evaluated eleven linear-chemistry atmospheric models of sulfur
deposition using 1980 observational data. This effort is noteworthy as
being the first organized effort at regional model evaluation. Not
only has the work of MD staff contributed to an increase in knowledge
on model performance, but also to new approaches to model evaluation.
Although evaluation methods for regional scale models which are
acceptable to all scientists have not been formally established, the
ISDME scientists have employed a careful system of statistical
procedures to arrive at their conclusions. The effort to rank the
models’ performance does not appear to aid the choice between models
for policy use and requires further research to be usable in that
context. In addition to the statistical evaluation, additional effort
is needed to outline conditions under which the models are or are not
applicable. For example, the simple Lagrangian models show the same
performance repeatedly over several years when annual averages are
compared. The ISDME analysis would be more useful if it had led to an
understanding of why the models show this type of performance. Use of
metrics based on spatial pattern is important, but requires
incorporation of spatial surface interpolation. Further research into
the applicabilities of kriging to surface concentration and deposition
fields is recommended.

Current in-house research focuses on design studies for surface
network, intra-spacing network and emissions data collection. The
design work relies heavily on kriging but does provide a basis for
quantitative design studies that focus on minimizing the evaluation
"error". The MD staff recognizes that kriging requires further
evaluations as a viable design and model evaluation tool and that other
evaluation methods should be explored.

13


TABLE OF CONTENTS

CHARGE TO REVIEW PANEL 2. 1 1 1 ee ee ee ee ee ee es
GENERAL OVERVIEW 2. 6. ee ee et
PROGRAM ASSESSMENT «ww ww ee ee ee ee ee et ee
Boundary Layer/Turbulence Research .. 1... +--+ ee
Regional Oxidant Modeling... 1... eee eee wee
Regional Particulate Modeling. ...... rn
Regional Acid Deposition Modeling Area. .....
Complex Terrain Dispersion Modeling ....... we
Wake Effects Research . 2... ee ee ee ee ee es
Technology Transfer/Application. .......-e e+e
APPENDIX A: Peer Review Panel . 2... 2... ee eee rrr
APPENDIX B: Agenda for the Peer Review of the In-House
Research by the Meteorology Division ..... we
APPENDIX C: Comments on Model Evaluation ..... 2+ +e ees
APPENDIX D: ASRL Process Evaluation for Meteorology
Division In-House Research Peer Review ......--
APPENDIX E: Results of the EPA Participant Survey
Meteorology Division... 1 ee ee eee ee ees
APPENDIX F: ASRL Staff Response to Reviewers’ Comments .....

APPENDIX G:

The Laboratory Director’s Review
Comments on the Panel Report and the ASRL
Staff Response . 2... 1 ee ee ee ee ee et


Results of the EPA Participant Survey
Meteorology Division
In-House Research Peer Review

The Atmospheric Sciences Research Laboratory of the US
Environmental Protection Agency convened a panel of scientific experts
to review the in-house research done by the Meteorology and Assessment
Division. EPA personnel involved in the program review were surveyed
as to the effectiveness and usefulness of the peer review process to
them; and to identify problem areas that needed improving. Thirteen
surveys were completed.

The evaluation instrument was designed to assess the following
aspects of the process: 1) review format and logistics; 2) review
panel; and 3) your assessment of the peer review process. A section
was also provided for comments and suggestions. The participants were
instructed to respond to 10 items by circling numbers from 1] to 4 (with
1 representing very satisfied; 2-satisfied; 3-dissatisfied; and 4-very
dissatisfied). Table 2 presents a summary of the ratings for these 10
items. None of the participants were “very dissatisfied" with any
items, although nearly all items received at least one "dissatisfied"
rating. However, the majority of the ratings were in the "satisfied"
and "very satisfied" categories.

34


01:20 - 01:35 PM

01:35 - 02:20 PM
02:20 - 02:40 PM

02:40 - 03:00 PM

03:00 - 03:30 M

03:30 - 04:00 PM

04:00 - 04:30 PM

04:30 - 05:00 PM

Wednesday, Sept. 16

08:00 — 08:20 AM
08:20 - 08:40 AM

08:40 - 08:50 AM

08:50 - 09:15 AM

09:15 - 09:40 AM

09:40 - 10:00 AM

10:00 - 10:30 AM

Relation of Error Bounds of
Maxmium Concentration Estimates
to Meteorology Uncertainty

Regional Oxidant Modeling
Regional Oxidant Model (ROM)
Biogenic Emissions Inventory
Development for Regional Oxidant
Modeling

Break

Ozone Dry Deposition Over Large
Areas from Aircraft Measurements

Verification Procedures Applied
to the Regional Oxidant Model

Evaluation of Regional Oxidant
Model

Executive Session for Review
Panel and MD Division Director

Regional Particulate Modeli
Regional Particulate Model (RPM)

Cloud Processes Module for
Regional Particulate Modeling

Utilization of RINEPH Data for
Regional Particulate Modeling

Sensitivity Analysis of MESOPUFF

II and Evaluation with CAPTEX Data

Regional Lagrangian Model of Air
Polution (RELMAP) Sensitivity
Analysis and Evaluation for
Particulate Matter

Coffee Break and Introductions of

Review Panel and ASRL Presenters

Regional Acid Deposition Modeling

International Sulfur Deposition
Model Evaluation (ISDME)

22

John Irwin

Bob Lamb

Jim Reagan

Jim Godowitch

Joan Novak

Ken Schere

Frank Schiermeier

John Clarke
Frank Binkowski

Russ Bullock

Jim Godowitch

Brian Eder

Terry Clark

10

14
15

16

17

18


U.S. ENVIRONMENTAL PROTECTION AGENCY
ATMOSPHERIC SCIENCES RESEARCH LABORATORY
PEER REVIEW AND WORKSHOP MANAGEMENT SERVICES

Contract Number 68-02-4129

Project Officer
Ronald K. Patterson

Prepared by

EX Research and Evaluation Associates, Inc.

1030 15th Street, NW., Suite 750
Washington, D.C. 20005
(202) 842-2200

727 Eastowne Drive, Suite 200A
Chapel Hill, N.C. 27514
(919) 493-1661


: We agree. Research on the study of diffusion in building wakes is
intended to provide a balance between the immediate needs for regulatory appli-
cation and the need to develop a solid research base. Improvements to the illu-
mination techniques have been made, and more thorough statistical evaluations of
the resulting data are to be pursued once the relation between video image
intensity and vertically integrated concentration is firmly established.

Comment, pp. 15-16: The work on auto exhaust dispersion is clearly associated
with EPA's need to model the impact of a major source of atmospheric pollution.
It is difficult therefore to judge what should or should not be done since the
work described was very incomplete in itself

Response: The presentation of the results from the study of dispersion in
automobile wakes was indeed only a part of a larger project but, as an example
of an in-house “wake effects" study, it was included in the peer review. No
further work in this area is planned for the near future.

Appendix C, pp. 27-28: Comments on Model Evaluation

Response: The four categories described by the peer review panel are the major
elements pertaining to application of a model to better understand and charac-
terize its behavior, performance and capabilities (as distinct from applications
of the model to answer a specific policy question). While we might quibble some
about the details of the descriptions under each category, possibly seeing them
a bit more interconnected than presented by the review panel, we tend to agree.

The main point is that a good evaluation requires a team with members who are
good at posing informative and useful tests of the model as well as members who
are good at quantifying and objectively interpreting those tests, the latter
usually being someone with statistical expertise. We believe that our present
actions are in concert with the panel recommendation. We have included persons
with statistical experience as part of the International Sulfur Deposition Model
Evaluation and have statisticians on the model evaluation planning team for the
Regional Acid Deposition Model (RADM) evaluation.

Reviewer Recommendations, p. 32: Provide organizational chart to all panel mem-
bers in advance of meeting. Also...it would have been useful to have hada few
pages describing the Meteorology Division's mission...

Response: These materials were indeed provided as indicated in the first
sentence on page 2 of the peer review report.

i mI : : Two of the reviewers seemed familiar with air pollu-
tion modeling and meteorology; the others were much less familiar with this sub-
field of meteorology. Although they may be professional meteorologists, that is
no guarantee they are knowledgable in all aspects of the field.

: Per my instructions to the ASRL Peer Review Coordinator, the panel
was constituted by representatives from the disciplines of statistics, microme-
teorology, systems analysis, fluid modeling, and meteorological modeling. This
mixture was needed to ensure that at least one reviewer was knowledgable in each
of the technical presentations.

46


ATMOSPHERIC SCIENCES RESEARCH LABORATORY
Process Evaluation for Meteorology Division
In-House Research Peer Review

The Atmospheric Sciences Research Laboratory (ASRL) of the US
Environmental Protection Agency convened a panel of scientific experts
on September 15 - 18, to review the in-house research of the
Meteorology and Assessment Division. The panel consisted of six
scientists. These reviewers were asked to evaluate the process
involved in preparing and implementing this specific meeting. Five
panelists completed the process evaluation.

The evaluation instrument was designed to assess the following
aspects of the process: 1) Preview Materials; 2) Process and
Logistical Information; and 3) the Review Meeting. A section was also
provided for reviewers to give their comments and recommendations. The
reviewers were instructed to respond to 15 items by circling numbers
from 1 to 5 (with 1 representing poor; 2-fair; 3-good; 4-very good; and
5-excellent).

Table 1 presents a summary of the reviewers’ rating for the 15
items. No items were rated poor. All items received some "excellent"
ratings except for overall peer review process and adequacy of time for
executive sessions. Most categories were either rated "good" or
"excellent". Three items were rated fair: adequacy of time available
to preview, meeting purpose, and overall peer review process. Thirteen
items were rated excellent: written quality, technical quality,
utility for outside reviewer, adequacy of time available to preview,
meeting purpose, scheduling of meeting, time/preparation requirement of
reviewers, timeliness of meeting notification, timeliness of logistical
information, adequacy of time for discussion with EPA staff, quality
and utility of presentations, quality and utility of materials
disseminated, and support services and activities.

30


Comment, pp. 4-5: One of the major challenges to ORD management would seem to
be finding and preserving a balance of staff time allocated to quick response
needs of the Agency on the one hand and longer-term needs on the other. Although
a good balance seems to have been achieved by the MD, it also seems to the panel
that ORD management in general has a responsibility to both the scientific staff
and to the Agency to take steps to guard against the erosion of the Agency's
research capability... A more active role by the MD is setting research priori-
ties would help to guard against the erosion of the MD's research capability by
an overload of quick response tasks.

Response: ORD management executes a series of steps each year in the budget
planning process that includes the ranking of existing research programs and new
research initiatives (prepared by Laboratory scientists). These priorities are
negotiated by research committees that are composed of both ORD and regulatory
program personnel. Unfortunately, this deliberate planning process is then
subject to the disruptive redirections described in the previous response. It
is thus up to us to sort out these conflicting priorities to achieve the "good
balance (that) seems to have been achieved by the MD."

Comment, pp. 4-5: More importantly, it should engender a vision and a coherent,
internally developed agenda for the MD's research. Even though the MD's tasks
are ultimately determined outside the Division, such an agenda would be an
important reference for establishing the MD's research priorities.

Response: We feel that we do have such an ongoing agenda for the Division's
research, and that this agenda is frequently used in sorting out conflicting
priorities when they arise.

Comment, p. 5: Finally, the emerging program concept of Advanced Research Model
Implementation is strongly endorsed by the panel

Response: The establishment of the ASRL Research Modeling Facility is our
attempt to prevent further erosion of the Division's research capability by not
allowing scientific expertise to be diluted by model applications at the expense
of ongoing model development and evaluation.

Comment, pp. 7-8: The rotary spectral density application in the analyses of
turbulence data is not obviously relevant to EPA objectives even though the
quality of the work is good. Such work may not enhance our understanding of
boundary layer turbulence significantly. The interaction with the outside com-
munity is not very apparent.

Response: We take strong exception to the implication of non-relevance. Dif-
fusion models, which are the Division's most important product, are basically
the application of turbulence theory to practical problems. Since the develop-
ment of turbulence theory (and its mathematical treatment) is far from "solved",
it is appropriate for us to spend some of our effort in refining the basis for
diffusion models. Whether the work will be successful or not is, of course, an
open question as it is with a great deal of true research. While it may not have
been made apparent to the reviewers, the spectral research has been reviewed
very favorably by personnel of the NOAA's Wave Propagation Laboratory.

39


02:50 - 03:10 PM

03:10 - 03:30 PM

03:30 - 03:50 PM

03:50 — 04:15 PM

04:15 - 04:45 PM

Friday, Sept. 18
08:30 - 10:00 AM.

10:10 AM

Comparing Air Quality with
Present or Proposed Regulatory
Standards

Rise and Dispersion of Merging
Buoyant Plumes in a Stratified
Crossflow (Ocean Outfalls)

Dispersion of Dense Gas Over a
Ramp

Advanced Model Operation and
Analysis

Ralph Larsen

Bill Snyder

Bill Snyder

Frank Schiermeier

Executive Session for Review Panel Frank Schiermeier

and MD Division Director

Reviewer Debriefing with ASRL
Director ©

Report Preparation

25

Al Ellison

43

44

45

46


Skill scores and/or performance evaluation may objectively rate the ROM
concentrations satisfactorily when instances of point concentrations
actually fail miserably. New methods of evaluation should be
considered that weigh both spatial and temporal scales or features.

If the ROM is labeled as a zero-order modeling effort, then its
performance will be properly evaluated. More support of this effort
should be provided. Also, apparently none of the field programs
to-date have provided high resolution (space and time) rawinsonde data
sets. Since a diagnostic model approach is acceptable, more emphasis
should be placed on collecting a more representative data set for model
development and performance evaluation.

ROM development apparently needs more computing cycles. Algorithms
for the model are not fully understood (for example, incorporation of
the effect of clouds and radiative processes). The ROM model should
interact more with numerical weather prediction model efforts. Can
some nested grid simulations and use of trajectory model forecasts (by
NOAA-NWS) be considered? Trajectory models combined with chemical
reactions code may yield some useful results.

In several instances the scientific capability of the involved
scientists exceeds the scientific credibility of ROM. For example, it
has been hypothesized that receptor sites can be grouped into classes,
such that concentrations averaged over all sites in a given receptor
class at a given hour is a quantity that is approximately the same for
all realizations of the concentration ensemble. If this is true, then
with respect to the receptor class average (RCA), the concentration
field is a quasi-deterministic variable. Therefore, model simulations
of single realizations can be interpreted in the conventional
deterministic manner. Although this hypothesis has not yet been
tested, it is assumed for the present purposes that it is correct. It
is unscientific to proceed in this way, though it may be politically
and practically sound.

Biogenic emissions inventory development is essential for the
application of ROM. Currently, the work involves translation of known
emission information to an hourly emissions ROM grid. This requires
allocation of land use and biomass information to grid cells,

9


Comments, pp. 32 and 36: Lack of adequate time for presentations.

Response: As noted in some of my preceding responses, the time allotments for
Presentations were occasionally too limited to accommodate complete descriptions
of individual in-house research efforts. For example, the Regional Oxidant Model
(ROM) and supporting components were presented in two and one-half hours, where-
as the ROM was allotted a full day in an ASRL project-oriented peer review in
January 1987, In adhering to this stringent schedule, therefore, presenters
were only able to describe the basic tenents of their research projects at the
expense of discussing ancillary topics such as relevancy to the Agency's mis-
sion, publication of results, and involvement with the scientific community at
large. The peer review report shows that, in most cases, the reviewers took
this limitation into consideration.

References.

Dahai, X., S. G. Perry, and P. L. Finkelstein, 1986: Multi-Scaling and Exponen-
tial Fitting in Autocorrelation Analysis. Extended Abstracts, Fifth Joint Con-
ference on Applications of Air Pollution Meteorology, AMS, pp. 348-351.

Hanna, S. R., 1981: Lagrangian and Eulerian Time-Scale Relations in the Daytime
Boundary Layer. Journal of Applied Meteorology, 20:242-249.

Hanna, S. R., 1986: Spectra of Concentration Fluctuations - The Two Time Scales
of a Meandering Plume. Atmospheric Environment, 20:1131-1137

Lamb, R. G. and S. K. Hati, 1987: The Representation of Atmospheric Motion in
Models of Regional~Scale Air Pollution. Journal of Climate and Applied Meteor-
ology, 26:837-846.

Misra, P. K., 1979: The Auto-Correlation Function of the Vertical Velocity in
the Low-Frequency Range. Preprints, Fourth Symposium on Turbulence, Diffusion,
and Air Pollution, AMS, pp. 41-45.

Moore, G. E., M. Liu, and L. Shi, 1985: Estimates of Integral Time Scales from
a 100-M Meteorological Tower at a Plains Site. Boundary-Layer Meteorology,
31:349-368.

Sutton, 0. G., 1953: Micrometeorology. McGraw-Hill Book Company, New York, NY,
333 pp.

Weber, A. H., J. S. Irwin, W. B. Petersen, J. J. Mathis, and J. P. Kahler, 1982:

Spectral Scales in the Atmospheric Boundary Layer. Journal of Applied Meteorol-
ogy, 21:1622-1632.

47


XD S74y,
s ®

UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
> ". ATMOSPHERIC SCIENCES RESEARCH LABORATORY

7a paoweTM RESEARCH TRIANGLE PARK
NORTH CAROLINA 27711

hacen

agony

DATE: February 1, 1988

SUBJECT: Meteorology Division In-house Program
FROM: Jack H. Shreffler ~}e“H1 She.
Deputy Director, ASRL/ (MD-59)

TO: Ronald K. Patterson
TPRO, ASRL (MD-59)

I have read the Peer Review report on the in-house research program
of the ASRL Meteorology Division and the response by its Director, Francis
A. Schiermeier. The report is one of the most complimentary I have re-
viewed and the Director's response is complete and adequate in all respects.
Moreover, the Summary of Process Evaluations by reviewers indicates a very
good preparation and execution of the peer review meeting, for which Research
and Evaluation Associates, Inc. and Project Officer Ron Patterson deserve
credit.

Management should not get involved in fixing things that are not broken.

cc: A. Ellison

49


industrious efforts to provide what is needed with inadequate resources
of manpower and computing facilities, resulting in contributions that
were incomplete. .

The cloud process module work may evolve into some good science.
However, this component of both the ROM and the RPM (or any regional
scale model) is not getting the attention it merits. The simplistic
approach presented is a reasonable application of the state-of-the-art
knowledge for simple cloud model formulation (primarily a thermodynamic
emphasis). Effort needs to be extended to properly treat the total
entrainment process between the cloud top and the overlying interfacial
layer. Cloud ensemble scale should be considered in accordance with
the ROM grid length.

The TINEPH data may be all right for radiation studies but its
value in the wet deposition and cloud processing modules is very
questionable. Efforts should be made to incorporate the GOES imagery,
which is very useful for the ROM domain. The work on this project
epitomizes the vitality offered by young new staff members.

The MESOPUFFII modeling effort is probably important in principle,
but (very good) performance evaluation shows poor performance. The
CAPTEX data set appears adequate to test performance. Best results
were obtained for cases of no directional shear. Work should be
published in reviewed journals (not just an EPA report).

The evaluations of the Regional Lagrangian Model of Air Pollution
RELMAP has served to expose NAPAP emission inventory deficiencies.
Thoroughness and comprehensive knowledge were evident in the
presentation.

Regional Acid Deposition Modeling Area

MD in-house research activities related to regional acid deposition
modeling involve the International Sulfur Deposition Model Evaluation
(ISDME) project, Regional Acid Deposition Model (RADM) evaluation and
assessment projects, and cumulus cloud venting based on data from
VENTEX, CUVENT, NEROS, and NASA Langley field experiments. The
in-house activities reviewed comprise only a fraction of the project

12