**NASA Advanced Satellite Aviation-weather Products (ASAP) Project**

**In-Flight Icing Benchmarking**

**Phase 1 Report**

**EXECUTIVE SUMMARY**

NCAR and NASA LaRC have undertaken an exercise to establish the accuracy
of their icing diagnosis products: the Current Icing Potential (CIP)
algorithm and the GOES Cloud Diagnosis Products. The purpose of the
exercise is to benchmark current capability so that future improvements
can be measured.

This report describes the first-year activities of the benchmarking
exercise. During Phase 1, products to be benchmarked were selected,
methods for verification were established, and initial verification was
completed. Case studies using in situ aircraft data for more detailed
comparison than can be accomplished by PIREP-based methods were selected
and begun.

The Phase 1 work described here shows that both the CIP-20 and GCDP
Cloud Phase product have skill in determining where inflight icing
conditions exist. The skill is quantified so that future improvements,
mainly incorporation of selected GCDP into CIP, can be monitored.

The work is not finished. As we enter into Phase 2 we will verify
additional GCDP (icing risk, liquid water path, etc.) and CIP-20 (SLD
and severity) outputs, and make direct comparisons between CIP and GCDP.
We will also tackle the problem of determination of cloud top height, as
we have found discrepancies between the two techniques. Case study
analyses will be expanded to more locations and projects in order to
better understand the capabilities of the algorithms and identify
potential areas of improvement.

Additionally, we will be examining the algorithms to see where they may
be improved or augmented using additional satellite channels. One or
more GCDP products will be chosen for incorporation into the CIP and
comparisons from the October 2003 -- March 2004 verification time period
will be done to gage improvements.

All in all, the results to date are promising in that both products show
skill in diagnosing icing conditions and we are beginning to understand
where meshing of the two will bring improvements. We have devised
methods for verification and comparison that can readily be adapted to
other CIP and GDCP outputs. Finally, we have established good
communications between NASA LaRC and NCAR which we hope to continue into
the future.

**INTRODUCTION**

This report summarizes the results of a benchmarking exercise conducted
as part of the Advanced Satellite Aviation-Weather Products (ASAP)
Program conducted by the National Aeronautic and Space Administration
(NASA) and the Federal Aviation Administration (FAA) Aviation Weather
Research Program (AWRP) to enhance and extend the use of satellite data
sets for applications in aviation weather.

The goal of ASAP is to increase and optimize the usage of satellite data
sets within the existing PDT structure and to transfer satellite
expertise to the PDTs. New satellite products or data sets would be
integrated into the PDT's suite of weather products and undergo the
normal FAA procedures for testing and transition to operational use.

ASAP enables collaboration between NASA and the FAA AWRP utilizing the
University of Wisconsin Cooperative Institute for Meteorological
Satellite Studies (UW-CIMSS) and the AWRP PDTs. This collaboration
involves assistance in testing and evaluation of existing satellite
algorithms developed or proposed by AWRP teams, the introduction of new
techniques and data sets to the PDTs from the satellite community, and
providing PDT access to satellite data sets available through CIMSS for
research and development purposes. New satellite information would be
integrated into the PDTs' suite of weather products that undergo
standard FAA procedures for testing and transition to operational use,
including implementation in official aviation weather providers such as
the National Weather Service.

The FAA's Aviation Weather Research Program (AWRP) formed Product
Development Teams (PDTs) with the responsibility to develop, test, and
transfer into operational use new weather products to improve the safety
and efficiency of the Nation's aviation system. The existing PDTs
already make use of satellite data, in combination with other data
resources such as radars, surface observing systems, numerical weather
prediction models, and pilot reports to produce integrated,
comprehensive products for the aviation community. Satellite remote
sensing capabilities, however, have advanced so rapidly that the PDTs
are not always using all available satellite data sets to an optimum
level. At the same time, a new generation of advanced meteorological
satellites is currently being developed that promise vastly enhanced
capabilities which have the potential to have a major impact on future
aviation weather products.

This benchmarking exercise is part of Phase I of ASAP which focuses on
use of *current* satellite data and processing techniques to address
various aviation problems (current includes new platforms to be launched
in the next year or two). We have concentrated our efforts on
observations from the Geostationary Operational Environmental Satellite
(GOES) East and West instruments (currently GOES-12 and GOES-10,
respectively).

Icing hazards are fundamentally dependent on the size and phase (ice or
water) of hydrometeors (including water drops and ice crystals)
throughout the depth of the cloud. There is a similar dependence on
precipitation type and amount for predicting the impact of winter storms
on aircraft operations in the airport terminal area. While
infrared-emission remote sensing techniques may limit sensing through
clouds, multi-spectral processing techniques have already shown skill in
monitoring hydrometeor size and phase information at cloud top. This
cloud top information can be valuable, but has to be used with caution
since the particles at cloud top may not be typical of the particles
below. For example, air motions at cloud top may produce liquid drops,
while underlying layers might be totally glaciated.

Extensive verification of basic satellite-derived products (i.e. total
precipitable water, cloud top heights, temperature and moisture
retrievals) has already been conducted using ground-based and aircraft
instrument measurements (Smith et al. 1996, Schmit et al. 2002).
Collaborations with the DOE ARM project have allowed climatological
verification of satellite radiances and value-added information. A
best-estimate product for temperature and moisture derived from DOE SGP
site Raman Lidar, AERI, microwave radiometer, and radiosonde information
is currently providing a way to evaluate EOS Terra (and soon EOS Aqua)
overpass radiance calibration. NASA EOS-sponsored field program
verification has provided the opportunity to collect data sets from a
variety of atmospheric conditions to provide direct radiance
verification of both POES and GOES satellite instrumentation.

Evaluation of icing parameters such as supercooled liquid water using
the current generation GOES instruments for the enhancement of
nationwide icing products. This will also involve the exploration of
cloud optical depths, hydrometeor size, phase and total water path in
near-real time. The methods of Smith et al. (2000a,b), similar to those
developed for use with MODIS data but which evaluate supercooled liquid
water using the current generation GOES instruments, can be also be
implemented. The Smith et al. techniques also provide cloud optical
depths, hydrometeor size, phase and total water path in near-real time.
These methods take advantage of GOES abilities to measure conditions
within the top several hundred meters of clouds over large geographical
regions.

The benchmarking was divided into three task areas described as follows.

Task 1: Icing PIREPs will be used to verify the accuracy of CIP and
satellite icing products using methods already established by the
IFIPDT.

Task 2: Recent field research program data will be used to examine
selected cases and provide insight on the behavior of the products.

Task 3: CIP output will be combined with one or more NASA satellite
products to simulate an integrated product and predict benefits.

These tasks are somewhat flexible; the goal is to provide a benchmark of
the capabilities of the NASA LaRC cloud products and CIP as well as to
assess possibilities for combining them to construct a superior icing
diagnosis product. Progress to date on these tasks is reported here.

**NASA LANGLEY SATELLITE ICING AND CLOUD PRODUCTS**

The Langley cloud products (Minnis et al. 2001) are derived from
half-hourly Geostationary Operational Environmental Satellite (GOES)
data taken from GOES-10 (West) and GOES-12 (East) using the Visible
Infrared Solar-infrared Split-window Technique (VISST) during the
daytime (Minnis et al. 1995, 1998). Each 4-km GOES pixel is first
classified as clear or cloudy using a complex cloud identification
scheme (Trepte et al. 1999). Each of the cloudy pixels are then analyzed
with the VISST to determine cloud phase, optical depth, effective
particle size, effective temperature, effective height, and ice or
liquid water path. These parameters are then used to estimate cloud-top
and base altitudes and temperatures. The analyses utilize the 0.65, 3.9,
10.8, and 12.0 µm channels available on most GOES satellites.

A prototype aircraft icing index (Minnis et al. 2003, 2004) was
developed based from initial comparisons of pilot icing reports and the
Langley cloud products (Smith et al. 2000, 2002, 2003) using
specifically the cloud temperature T~c~, cloud optical depth , cloud
phase, cloud droplet effective radius re, and the liquid water path LWP.
The prototype criteria are summarized in Table 1. The logic tree first
checks whether the pixel is clear or not. If clear, it is eliminated
from further consideration. If cloudy, it is eliminated from further
processing if the T~c~ \> 272 K (too warm for icing) or if it is
classified as an ice cloud with \< 8. This latter criterion was
established based on the logic that the optical depth would be larger if
a significant water cloud exists beneath the ice cloud. Any water cloud
below the ice cloud is probably too thin to induce icing. The remaining
pixels are then examined for the potential to cause icing.

If the observed cloud is classified as ice and it has a large optical
depth ( \> 8), it is assumed that the potential for a significant water
cloud exists underneath the ice cloud, but it is not possible, at this
point, to determine whether it is there or not. The pixel is therefore
classified as unknown or indeterminate. It will require further study or
combination with the other CIP data to arrive at a firmer
classification. The remaining pixels are all classified as supercooled
water clouds. They are classified as having no, low, middle, or high
icing potential based on the values of *r~e~*, *T~c~*, and *LWP* as
outlined in Table 1. Any supercooled water cloud pixels having
characteristics that do not meet any of the criteria 2 - 9 in Table 1
are not considered as icing threats. The threshold values were developed
from weak relationships developed from previous PIREPS comparisons. An
example of the icing product is shown in Figure 1 for a day in March
2004. Areas with icing potential are apparent over Iowa, western
Tennessee, northern North Dakota, around North Carolina, and in the
Pacific Northwest. Many areas of indeterminate icing are found as a
result of extensive high cloud cover. Some small areas of potential
icing are seen around the edges of some cirrus clouds as a result of
some overlapping conditions.

This initial benchmarking effort will compare the icing conditions
derived from the Table 1 criteria to those from PIREPS obtained during
2003 and 2004 and to the current CIP. The results of this benchmarking
effort will be used to adjust the thresholds to obtain the optimal
values for operational use of the cloud products to determine icing
potential and to develop logic to further account for cloud overlap
effects.

**Table 1. Prototype icing classification criteria for NASA LaRC
satellite-based products.**

  ------------ ------------ ---------------- ------------ ----------------
  Value        criteria                                   icing intensity

  0            clear or                                   none
               water cloud                                
               w/ T~c~ \>                                 
               272 K or LWP                               
               \< 100                                     
               gm^-2^ or                                  
               ice cloud                                  
               with \< 8)                                

  1            ice cloud                                  unknown
               with  \> 8                                

               r~e~ (µm)    LWP (gm^-2^)     T~c~ (K)     

  2            \> 10        \> 100           \< 272       low

  3            \> 10        \> 200           \< 272       mid

  4            \> 10        \> 300           \< 272       high

  5            \> 8         \> 400           \< 272       low

  6            \> 8         \> 500           \< 272       mid

  7            \> 10        \> 300           \< 253       high

  8            \> 8         \> 400           \< 253       high
  ------------ ------------ ---------------- ------------ ----------------

![](media/image1.jpeg){width="5.33125in" height="6.377083333333333in"}

> *Figure 1. Cloud and icing data for 1915 UTC, 15 March 2004. (a)
> Stitched GOES-10/12 infrared brightness temperature images. (b) Icing
> categories.*

**THE CURRENT ICING POTENTIAL (CIP) PRODUCT**

Icing has been a critical factor in numerous fatal aircraft accidents,
and can disrupt domestic flight operations. Current weather products do
not adequately address these dangerous and disruptive events. Avoidance
of these events requires accurate, high-resolution, operational
forecasts of atmospheric icing conditions. The goal is an hourly gridded
depiction or forecast of in-flight icing based on operational model
output combined with real-time sensor and remote sensing data, including
icing characteristics (severity and type). The philosophy is to minimize
volume warned while not compromising the probability of detection.

The first In-Flight Icing PDT product to be transferred to operational
use in the National Weather Service was the Current Icing Potential
(CIP, see McDonough et al., 2000 and Bernstein et al. 2004). A
subsequent product, the Forecast Icing Potential (FIP) has also been
developed and is in an experimental status. The CIP and FIP algorithms
apply fuzzy logic techniques to combine up to 56 interest fields into
one fused product. CIP presently combines data from five
sources--multispectral GOES imagery, model output from the RUC model,
surface observations, NEXRAD radar data, and pilot reports. The CIP and
FIP are available on the ADDS web page at:
<http://adds.aviationweather.noaa.gov/projects/adds/icing/> *(*see
example plot in Fig. 2)

![](media/image2.png){width="6.502083333333333in"
height="4.136111111111111in"}

> *Figure 2: Example of CIP hourly output. This example shows the
> maximum icing potential value in any 20-km gridded RUC column, for
> 1800 UTC, 14 November 2003. Letters are icing PIREPS; R=rime; C=clear;
> X=mixed and U=unknown ice type. Size of letters is related to reported
> severity.Scale is shown at bottom of figure.*

**Verification Methods**

The CIP verification was accomplished by evaluating the icing potential
field versus pilot reports (PIREPs) of positive and negative icing. Each
PIREP was matched to the closest CIP grid point and flight level. The
four grid points surrounding the observation as well as 1,000-ft[^1]
flight levels above and below the PIREP were examined. Currently, CIP
incorporates information from PIREPs in the hour prior to the forecast
time. Thus, this analysis only used observations in a time window of one
hour following the forecast valid time. Statistics were then computed
and analyzed. The verification statistics for CIP were then compared to
the Airmens' Meteorological Advisories (AIRMETs). Although the AIRMETs
are generally different than CIP (e.g., they are limited to a certain
volume and are intended to depict icing over a six hour period), the
comparison is included because AIRMETs are a readily available
operational icing forecast produced by forecasters at the Aviation
Weather Center (AWC).

The verification methods utilized in the evaluation of CIP are based on
standard verification concepts that recognize the underlying framework
for forecast verification and the associated high dimensionality of the
verification problem. The methods used are described in greater detail
in Brown (1996), Brown et al. (1997). The icing forecast verification
methodology outlined by Brown et al. (1997) treats icing forecasts and
observations as Yes/No values. Brown et al. (1999) outlines how this
method can be extended to forecasts with values on a continuous scale.
Specifically, icing diagnoses produced by CIP can be converted to a set
of Yes/No values by applying a variety of thresholds. For example,
applying a threshold of 0.30 to CIP diagnoses would lead to a Yes value
for all grid points with an icing potential value greater than or equal
to 0.30 while each grid point with a value less than 0.30 would be
assigned a NO value. The verification methods are based on the Yes/No
two-by-two contingency table (Table 2). Each cell in this table contains
a count of the number of times a particular forecast/observation pair
was observed. The counts on this table are observation-based (i.e., the
sum of the counts is the total number of Yes and No PIREPs over the
given time period) and therefore not all CIP grid points are
represented.

**Table 2: Contingency table for evaluation of dichotomous (e.g.,
*Yes/No***) forecasts. Elements in the cells are the counts of
forecast-observation pairs.

+----------------+--------------+--------------+---------------------+
| >              | > ***Obs     |              | > ***Total***       |
| ***Forecast*** | ervations*** |              |                     |
+----------------+--------------+--------------+---------------------+
|                | > ***Yes***  | > ***No***   |                     |
+----------------+--------------+--------------+---------------------+
| > ***Yes***    | > YY         | > YN         | > YY+YN             |
+----------------+--------------+--------------+---------------------+
| > ***No***     | > NY         | > NN         | > NY+NN             |
+----------------+--------------+--------------+---------------------+
| > ***Total***  | > YY+NY      | > YN+NN      | > YY+YN+NY+NN       |
+----------------+--------------+--------------+---------------------+

PODy and PODn are the primary verification statistics that are included
in this evaluation. They are estimates of the proportions of Yes and No
observations that were correctly diagnosed. Together, PODy and PODn
measure the ability of the forecasts to discriminate between Yes and No
icing observations. Table 3 gives the definition and description of
these statistics.

**Table 3: V**erification Statistics used in evaluation of CIP.

  ----------------- ------------------------- -----------------------------
  ***Statistic***   ***Definition***          ***Description***

  **PODy**          YY/(YY+NY)                Probability of detection of
                                              "*YES*" observations

  **PODn**          NN/(YN+NN)                Probability of Detection of
                                              "*No*" observations
  ----------------- ------------------------- -----------------------------

The relationship between PODy and 1-PODn for different thresholds is the
basis for the verification approach known as "Signal Detection Theory"
(SDT). This relationship can be represented for a given algorithm with
the curve joining the (1-PODN, PODy) points for different thresholds.
The resulting curve is known as the "Relative Operating Characteristic"
(ROC) curve in SDT. The area under the curve is a measure of overall
forecast skill and provides another measure that can be compared among
forecast products. This measure is not dependent on the threshold used.
A forecast with zero skill would have an ROC area of 0.5 or less.

***CIP Data Used for Benchmark Verification***

The valid time period for this evaluation was 01 October 2003 to 31
March 2004. The times compared were the 1500 UTC and 2100 UTC valid
times; a total of 341 CIP files were available for verification. All
available AIRMETs valid for the specific CIP times were scored. The
total number of PIREPs used as observations in the evaluation of CIP and
the AIRMETs is listed in Table 4.

**Table 4: Observations included in the CIP verification.**

  ------------------------------------- ---------------------------------
  ***Observation***                     ***Number of reports***

  **NO**                                10713

  **YES\[moderate or greater(mog)\]**   28316

  **YES(severe)**                       8005
  ------------------------------------- ---------------------------------

***Results***

CIP is relatively efficient at detecting icing conditions of detection
for MOG YES reports \[PODy(MOG) of 0.74-0.82, see Fig. 3\], and a
corresponding probability of detection of NO reports (PODn) between
0.62-0.68, depending on the CIP threshold used. AIRMETs detected a
similar number of YES reports with a \[PODy(MOG)\] of 0.751 also
capturing a similar number of NO reports with a PODn of 0.635. Figure 4
is a ROC plot of all of the CIP thresholds for all of the valid times
with the statistics for the AIRMET overlaid. It is a sufficient
depiction of how skillful the CIP icing potential field is compared
AIRMETS.

![](media/image3.png){width="4.427777777777778in"
height="3.0770833333333334in"}

> *Figure 3: Comparison of PODy(mog) vs. 1-PODn for CIP over winter 2003
> and for times valid for Benchmarking period (Oct'03 -- Mar'04)..*

![](media/image4.png){width="4.427777777777778in"
height="3.154166666666667in"}

*Figure 4:. Same as Fig. 3, except for CIP and AIRMETs.*

**NASA LaRC GOES-DERIVED CLOUD PRODUCTS (GDCP) VERIFICATION**

The verification study for the GDCP will first be done using PIREPs of
positive and negative icing and is very similar to the CIP verification.
The verification methods will be essentially the same, but some
variation is inevitable because of the differences in the products
themselves. While CIP is a three-dimensional product, able to be matched
to any PIREP in its domain, the GDCP are two-dimensional and can only be
considered valid near cloud top. The GDCP also have a finer horizontal
resolution than CIP that must be accounted for. These differences and
their resolution will be discussed further in the next section.

# *Methods*

To ensure a fair verification and allow established techniques to be
used the products must first be put on the same grid. The CIP is run on
the 20 km Rapid Update Cycle (RUC) model grid. The GDCP are on the same
grid, but the horizontal resolution is 5 km. Using the same grid and
resolution for both products makes sure that the satellite products
aren't penalized for the increase in horizontal resolution. For example,
a PIREP will be matched to the nearest four CIP grid points (see section
on CIP verification), each 20-km square, and will cover roughly 400 km2
of area. If this same approach were used for the GDCP (5-km horizontal
grid) only about 25 km2 of surface area would be examined. Each CIP grid
point would then contain 16 satellite grid points. The satellite
information from the 16 grid points was used to determine values
representative of the entire 20 km grid point. These values were saved
and became the basis of comparison to the PIREPs.

It was decided that the first field to be verified would be the cloud
phase product (see above for a description) because it could easily be
correlated to the CIP icing potential. If the phase of the cloud
diagnosed at any given pixel is liquid with temperature \< 0 °C one can
assume that there is the potential for icing there.

The following procedure was proposed in order to save as much
information as possible from the three datasets (PIREPs, CIP, and GDCP)
and to be able to change the verification easily if different
specifications were decided on. A database will be created that contains
an entry for every PIREP that matches the valid times between the start
and end date. Table 5 shows the columns of information that each entry
will have associated with it. Perhaps the biggest reason for this method
is the fact that the GDCP are two-dimensional. By saving things such as
the PIREP height and cloud top information from the CIP and GDCP various
methods for determining whether a PIREP was near enough to cloud top to
be included in the verification could be tried. If it is determined that
any PIREP within 1 km of a cloud top will be included then what should
be used as the cloud top for a given location? It might be best to look
at median values for all cloudy points or at the range of cloud top
heights for supercooled liquid water points only. Statistics can be
calculated from the information in the database that will give the most
consistent method for determining the best cloud top height.

Once this has been determined then the verification will be similar to
that for the CIP. If a PIREP with positive icing is near cloud top then
it will be compared to the four surrounding grid points. If any of those
grid points contains supercooled liquid water then that will be
considered a hit for the GDCP and vice versa.

***Data***

The valid time period for this evaluation is the same as for the CIP ---
01 October 2003 to 31 March 2004 with valid times of 1445 and 2045 UTC
(the nearest match to 1500 and 2100 UTC). A total of 316 GDCP files were
available for verification with each file representing a combination of
GOES-10 and GOES-12 data sets that have been stitched together at
longitude 99.5° W.

Because the cloud phase observation is only valid at cloud top, eight
different methods were employed in an attempt to discover the best way
to determine whether or not a PIREP was close enough to the defined
cloud top height. Each method was assigned a flag value and are
described in Table 6. This table shows the counts for the numbers of
PIREPs located within certain distances of the GDCP Cloud Top Height
(CTH) measurements. These different groups of obser-vations are used to
perform separate analyses on GDCP.

> **Table 5:** Information saved for each PIREP in the database. SLW
> (supercooled liquid water) phase represents cloudy points where the
> phase is liquid with T \< 0 °C.

  -----------------------------------------------------------------------
  **PIREP Information:**

  Date

  Time

  Latitude

  Longitude

  Aircraft Type

  Icing Base

  Icing Top

  Icing Intensity

  Icing Type

  Cloud Top

  **CIP Information (from the nearest 4 grid points):**

  Date of Run

  Time of Run

  Max Icing Potential

  Median Icing Top Height

  Max Icing Top Height

  Min Icing Top Height

  Median Cloud Top Height

  Max Cloud Top Height

  Min Cloud Top Height

  Median Cloud Top Temperature

  Max Cloud Top Temperature

  Min Cloud Top Temperature
  -----------------------------------------------------------------------

  -----------------------------------------------------------------------
  **GDCP Information (from the nearest 4 grid points):**

  Date of Product

  Time of Product

  Number with Phase = SLW Cloud

  Number with Phase = Any other Cloud

  Number with Phase = Clear

  Number with Phase = Missing or Bad Data

  Median Cloud Top Height (All Cloudy Points)

  Max Cloud Top Height (All Cloudy Points)

  Min Cloud Top Height (All Cloudy Points)

  Median Cloud Top Height (SLW Points)

  Max Cloud Top Height (SLW Points)

  Min Cloud Top Height (SLW Points)

  Median Effective Temperature (All Cloudy Points)

  Max Effective Temperature (All Cloudy Points)

  Min Effective Temperature (All Cloudy Points)
  -----------------------------------------------------------------------

> **Table 6: Numbers of observations used in GDCP verification**

  ------------ ---------------------------- ------------- ------------------
  ***FLAG***   **All PIREPs within ...**    ***YES\       ***NO***
                                            MOG***        

  **1**        **+/- 1,000 ft of median     **134**       **1620**
               CTH\                                       
               (All Cloudy pixels)**                      

  **2**        **+/- 3,000 ft of median     **349**       **3936**
               CTH\                                       
               (All Cloudy pixels)**                      

  **3**        **+/- 1,000 ft of median     **130**       **1059**
               CTH\                                       
               (All SLW pixels)**                         

  **4**        **+/- 3,000 ft of median     **328**       **2790**
               CTH\                                       
               (All SLW pixels)**                         

  **5**        **+/- 1,000 ft of min/max    **643**       **6244**
               CTH\                                       
               (All Cloudy pixels)**                      

  **6**        **+/- 3,000 ft of min/max    **828**       **8229**
               CTH\                                       
               (All Cloudy pixels)**                      

  **7**        **+/- 1,000 ft of min/max    **393**       **3300**
               CTH\                                       
               (All SLW pixels)**                         

  **8**        **+/- 3,000 ft of min/max    **552**       **4827**
               CTH\                                       
               (All SLW pixels)**                         
  ------------ ---------------------------- ------------- ------------------

***Results***

The overall results are diagrammed in Figs. 5 and 6. Figure 5 is an ROC
diagram of CIP, AIRMET, and GDCP for Flag 1. As shown in Figs. 2 and 3,
CIP shows good skill with a large area under the ROC curve. The single
AIRMET data point is located just below the CIP line. The GDCP (cloud
phase) has positive area under the curve and thus, positive skill, but
has less area than the CIP. Figure 6 shows results for an evaluation
similar to that shown in Fig. 5 but for Flag 2. The results are similar
in that CIP and the AIRMET show more skill than GDCP. In these two
figures the PODy(MOG) values show that GDCP has some success in
diagnosing the positive icing areas, assuming that the presence of SLW
shows a potential for icing. The low PODn values are likely the reason
for the poor skill of GDCP using all eight thresholds.

> ![](media/image5.png){width="2.9277777777777776in"
> height="2.082638888888889in"}
>
> *Figure 5: Comparison of PODy(MOG) vs. 1-PODn for CIP, AIRMETs, and
> GDCP using reports +/‑ 1,000 ft of the GDCP median CTH value for all
> cloudy pixels.*
>
> ![](media/image6.png){width="2.926388888888889in"
> height="2.061111111111111in"}
>
> *Figure 6: Comparison of PODy(MOG) vs. 1-PODn for CIP, AIRMETs, and
> GDCP using reports +/‑ 3,000 ft of the GDCP median CTH value for all
> cloudy pixels.*

Table 7 shows the verification results using specific thresholds for
both CIP (0.05 and 0.15) and GDCP for all flag values (see Table 6).
These are compared with the results for the AIRMETs. The results show
that the PODy(MOG) values for GDCP ranged from 0.578 to 0.822 and were
somewhat comparable to the CIP (0.744 and 0.828) and the AIRMETs
(0.751). The PODn results are not as remarkable as the PODy(MOG) results
with scores ranging from 0.195 to 0.536 versus CIP (0.604 and 0.671) and
the AIRMETs (0.635). The low PODn for GDCP scores are also responsible
for bringing down the True Skill Statistics (TSSs), which ranges from
0.002 to 0.206 compared to CIP (0.432 and 0.415) and the AIRMETs
(0.386). Overall, the evaluation using GDCP-1 and GDCP-2 (Table 7) shows
the best results for the GDCP with TSSs of 0.178 and 0.206 respectively.

The poor results for the GDCP for PODn can be attributed to the nature
of the product and the verification methods. Since the GDCP product is
two-dimensional and is only valid near cloud-top, a smaller set of
observations was used for verification. The area that was verified
contained SLW and an assumption of positive icing potential was made in
order to obtain a verifiable field. The problem with this assumption is
that the presence of SLW does not necessarily imply that the area has
icing. In the case where SLW is present but icing is not, a negative
report of icing was recorded as a miss. Another apparent problem with
the verification method is the lack of an exact CTH measurement. If a
median cloud-top height measurement is too high and there is a negative
report of icing in an area just above the real cloud top containing SLW,
then a miss is recorded.

With the verification limited only to the areas around the GDCP CTH
measurement, the low values of PODn are not surprising. After all, the
GDCP only attempts to detect areas of positive SLW and thus positive
icing potential. If clear areas are added to the verification (i.e.
areas above the maximum CTH measurement) then the PODn results would
likely improve. However, addition of the negative icing areas would most
likely have a negative effect on the PODy(MOG) statistics.

Another potential problem with this verification approach is the
difficulty in evaluating a high-resolution observational product using a
low-resolution verification data set (PIREPs in this case). While this
can, in part, be compensated for by trying to combine many of the
high-resolution data pixels into larger spatial blocks, this approach
will necessarily reduce the PODn, as we have seen in this analysis.

> **Table 7**: Statistics for CIP, AIRMETs, and GDCP at Specific
> Thresholds.

  ----------------- ------------------------ --------------- ------------
  ***Product***     ***PODy(MOG)***          ***PODno***     ***TSS***

  ***CIP* (.05)**   **0.828**                **0.604**       **0.432**

  ***CIP* (.15)**   **0.744**                **0.671**       **0.415**

  ***AIRMET***      **0.751**                **0.635**       **0.386**

  ***GDCP-1***      **0.679**                **0.527**       **0.206**

  ***GDCP-2***      **0.642**                **0.536**       **0.178**

  ***GDCP-3***      **0.708**                **0.294**       **0.002**

  ***GDCP-4***      **0.741**                **0.306**       **0.047**

  ***GDCP-5***      **0.579**                **0.481**       **0.060**

  ***GDCP-6***      **0.578**                **0.499**       **0.077**

  ***GDCP-7***      **0.822**                **0.195**       **0.017**

  ***GDCP-8***      **0.790**                **0.241**       **0.031**
  ----------------- ------------------------ --------------- ------------

***Plans for Further Analyses***

In addition to cloud phase, the GDCP include a number of additional
products that may be of use in CIP and which will be verified as these
studies progress. These products include icing risk, liquid water path,
water drop radius, optical depth, cloud top pressure, and the cloud base
and top heights. It is likely, however, that a different sort of
verification will be needed to judge the effective-ness of some of these
products; possibly using PIREP reported icing intensity and/or research
aircraft data.

In addition to the PIREP-based verification, we also plan to conduct a
direct comparison of CIP and selected GDCP outputs including the total
areas covered as well as overlapping and non-overlapping areas. In
addition, we hope to derive statistics such as efficiency (POD divided
by total area) to help us determine how the GDCP can best be
incorporated into CIP.

**CASE STUDIES**

Under Task 2 of the Benchmarking exercise, recent field research program
data is used to examine selected cases for comparison with NASA LaRC
GOES-derived Cloud Products (GDCP) and Current Icing Potential (CIP)
product. Field programs identified for this exercise include
AIRS-II/THORPEX in the northeast U.S. and eastern Canada, NASA-Glenn
icing research flights in the Cleveland area, and WISP04 in the Colorado
Front Range region. Flight days where supercooled liquid was observed by
research aircraft were selected to assess the capability of the GDCP and
CIP to diagnose icing conditions. For completeness, cases without
supercooled liquid (i.e., glaciated clouds) were also included in the
analyses. It should be noted that each of these field programs occurred
recently (late 2003 -- early 2004), so aircraft data sets are in
preliminary form and may undergo modification with subsequent data
processing.

***Data Sets and Procedures***

Specific cases from the three field programs are described in Table 8.
Data from each flight were examined to ascertain times and locations of
cloud-top penetrations by the aircraft. These data points are most
suitable for comparison with the GDCPs since some of the
satellite-derived fields are two-dimensional, representing conditions at
cloud top. There is some uncertainty associated with identification of
cloud top penetration because an overlying cloud layer may have existed
without being observed by the aircraft. For this reason, it is sometimes
the case that the satellite products will not represent the same cloud
layer observed by the aircraft.

**Table 8: Cases selected for comparison of aircraft data with GDCP and
CIP.**

  ---------- ------------------------------- --------------- --------------
  **Case**   **Project**                     **Aircraft**    **Date**

  1          AIRS-II                         Convair         11-Nov-03

  2          AIRS-II                         Twin Otter      11-Nov-03

  3          AIRS-II/THORPEX                 Citation        24-Nov-03

  4          AIRS-II/THORPEX                 Citation        28-Nov-03

  5          AIRS-II/THORPEX                 Citation        30-Nov-03

  6          AIRS-II                         Twin Otter      30-Nov-03

  7          NASA-Glenn Icing Research       Twin Otter      27-Jan-04

  8          WISP04                          Citation        10-Mar-04
  ---------- ------------------------------- --------------- --------------

Cloud top penetrations by the aircraft were identified using available
time series data from each flight. Measurements of liquid water content,
cloud drop size distribution, and humidity are used in combination to
estimate the cloud top location. Flight notes compiled by mission
scientists are also used when available. Resulting estimates of cloud
top locations for each case are listed in Table 9. Values of variables
to be compared with GDCP and CIP output are extracted for each cloud top
penetration.

**Table 9: Cloud Top Penetrations for Cases Listed in Table 1**.

  ------------- ------------- ------------- ------------- ----------------
  **Case**      **Time        **Altitude    **Phase**     **Temperature
                (UTC)**       (m)**                       (ºC)**

  1             1933          3300          liquid        -4.9

  1             1944          5454          mixed         -19

  1             2020          3757          liquid        -7

  2             1933          3700\*        mixed         -5

  3             1956          10708         ice           -52

  4             1711          7515          ice           -28

  4             1818          8160          ice           -38

  4             2001          8691          ice           -56

  5             1748          2400          liquid        -11

  5             1835          2000          liquid        -12

  5             1855          2000          liquid        -12

  5             1922          2000          liquid        -12

  6             1924          2180          liquid        -8

  6             1943          2454          liquid        -10

  7             1604          1806          liquid        -3.8

  7             1622          2024          liquid        -4.7

  7             1655          2151          liquid        -5.5

  8             2230          3545          liquid        -6.5
  ------------- ------------- ------------- ------------- ----------------

Corresponding GDCPs from the NASA LaRC archives were obtained for each
aircraft cloud top penetration. Products derived from GOES-12 for the
ATREC domain were used for most of the cases. Products for one case not
covered by this domain were obtained from the GOES-10 CONUS data set.
These products are available at 30-min intervals, providing good close
temporal correspondence with aircraft observations in all cases. Using
latitude-longitude coordinates of the location where the aircraft
penetrated cloud top, GDCP files were searched for the nearest pixel.
Values of pertinent variables were extracted at the nearest pixel and
from adjacent pixels. Specifically, cloud thermodynamic phase, liquid
water path or ice water path, cloud effective temperature, and cloud top
height estimated from the VISST algorithm were obtained.

Similarly, CIP output was obtained for each case study. For the location
of aircraft cloud top penetration, icing potential was extracted at
aircraft-estimated cloud top height. CIP also estimates cloud top
height, so icing potential was obtained at the CIP-estimated altitude as
well.

***Aircraft vs. Satellite (GDCP***)

In this preliminary analysis, comparisons are made between variables
that are easily derived from each data source. These include cloud
thermodynamic phase, cloud top height, and cloud top temperature from
aircraft and GDCP (i.e., the GDCP temperature product is actually a
cloud effective temperature, but this variable represents cloud top
temperature well in optically thick clouds). Cloud top height as
estimated by CIP is also compared to each of the other data sources.
Icing potential as estimated by CIP cannot be compared directly to any
variable derived from the other sources, but is given in tabular form,
along with satellite and aircraft cloud phase estimates.

Cloud thermodynamic phase as estimated from aircraft measurements and
the GDCP's are compared in Figure 7. Qualitative assessment of the phase
observed by aircraft is converted into the phase categories used for the
GDCP. The categories that occur in these case studies are:

> 1 = liquid
>
> 2 = ice
>
> 4 = clear

When mixed phase conditions are observed by the aircraft, a value of 1.5
has been assigned. All of the clouds considered here have temperatures
below 0ºC at cloud top, so any occurrence of liquid in the data implies
supercooled liquid.

Of the 19 cloud top penetrations listed in Table 9, liquid phase clouds
are estimated by both aircraft and GDCP in 10 of the cases (Category 1).
In these cases, estimated cloud top heights are between 2000 and 4000 m,
and cloud top temperatures are in the range of -3.8 to -18ºC. In 3 of
the 18 cloud top penetrations, both data sources suggest ice clouds
(category 2). Higher cloud top heights (4500 to10700 m) and colder
temperatures (-13 to -56ºC) characterize these cases. In two cases, the
aircraft data suggested mixed phase conditions while the GDCP estimates
ice. In the 4 remaining cloud top penetrations, the aircraft and GDCP
estimates do not agree. In two of these cases, the GDCP estimated clear
skies (category 4) while the aircraft observed cloudy conditions (a
liquid phase cloud on Mar 10, 2004 and an ice phase cloud on Nov 11,
2003). Examination of the GDCP on these dates shows variability in the
cloud field over the region of interest. For example, conditions were
predominantly cloudy on Mar 10, 2004, but the satellite pixel nearest
the aircraft cloud top penetration happened to be clear. This result
suggests that averaging the GDCP values over a larger geographic area
might be more appropriate than averaging over the nearest pixel and
adjacent pixels. Finally, there are two cases where the aircraft
measurements suggest liquid phase and the GDPC estimates ice phase. In
both of these cases (and in one of the cases where mixed phase clouds
were observed by aircraft), there is a signifigant discrepancy between
cloud top heights and temperatures estimated by the aircraft and GDCP.
The higher, colder cloud tops estimated by GDCP suggest that the
aircraft actually penetrated a lower cloud layer, so the two data sets
were not portraying the same cloud.

The comparison of aircraft- and GDCP-estimated cloud top heights is
shown in Figure 8. In general, the aircraft cloud top height
measurements tend to be lower than the GDCP estimates. As noted
previously, this difference is partly attributable to the fact that the
aircraft may not have been measuring the highest cloud layer (i.e., the
cases where aircraft cloud top height is 3000-4000 m and GDCP cloud top
height is 7000-9000 m are probably examples of this situation). A
cluster of points with lower cloud top heights consistently shows higher
values in the GDCP (3000-5000 m) than observed by the aircraft
(1800-2900 m). Reasons for this discrepancy are not yet understood.

Cloud top temperatures are compared in Figure 7. The largest
discrepancies occur when aircraft temperatures are relatively warm (-4
to -10C) while corresponding GDCP temperatures are very cold (-30 to
-50C). These are the same cases where a large difference in cloud top
height estimates was detected, so the differences are likely due to
misidentification of true cloud top by the aircraft. In most other
cases, aircraft and GDCP temperatures are within 10C of each other. In
general, the aircraft temperatures are slightly higher which is
consistent with the smaller cloud top heights estimates noted above.

![](media/image7.wmf){width="3.611111111111111in"
height="4.003472222222222in"}

> *Figure 7: Comparison of cloud thermodynamic phase as derived from
> aircraft data and GDCP. The number of cases represented by each point
> is shown in parentheses.*
>
> ![](media/image8.wmf){width="3.620138888888889in"
> height="3.814583333333333in"}
>
> *Figure 8: Comparison of cloud top height as derived from aircraft
> data and GDCP.*
>
> ![](media/image9.wmf){width="4.502083333333333in"
> height="3.7694444444444444in"}*Figure 9: Comparison of cloud top
> temperature as derived from aircraft data and GDCP.*

***Aircraft vs. CIP***

Output from CIP is reported as icing potential, so a direct comparison
of phase estimates between aircraft data and CIP is not possible.
Instead, Table 8 lists the estimated cloud phase based on aircraft
observations together with the icing potential at the aircraft-estimated
cloud top height (note that the CIP cloud top height estimate may
differ). A positive icing potential value exists in all but one case
where the aircraft observed supercooled liquid. Icing potential was zero
in all cases where the aircraft observed ice phase at cloud top.

Comparison of cloud top height estimated by aircraft and CIP is shown in
Figure 10. Aircraft estimates of cloud top height are generally lower
than those produced by CIP. Reasons for this bias are under
investigation.

**Table 10**: Comparison aircraft-estimate cloud phase and Current Icing
Potential (CIP)

  ------------------------ ----------------------------------------------
  **Aircraft-estimated     **Current Icing Potential (CIP) nearest
  phase at cloud top**     aircraft cloud top altitude**

  liquid                   1.00

  liquid                   0.85

  liquid                   0.96

  liquid                   0.84

  liquid                   1.00

  mixed                    0.31

  liquid                   0.75

  mixed                    0.75

  ice                      0.00

  ice                      0.00

  ice                      0.00

  ice                      0.00

  liquid                   0.44

  liquid                   0.44

  liquid                   0.44

  liquid                   0.04

  liquid                   0.00

  liquid                   0.31

  liquid                   0.26
  ------------------------ ----------------------------------------------

> ![](media/image10.wmf){width="3.658333333333333in"
> height="4.113888888888889in"}*Figure 10: Comparison of cloud top
> height derived from aircraft data and CIP*

***Future Work on Case Studies***

A preliminary comparison of the basic variables available in GDCP, CIP,
and aircraft data sets has been conducted for several readily available
data sets. Although reasonable agreement exists between phase estimates,
a full investigation of the conditions under which the GDCP and CIP
estimates are likely to be accurate or inaccurate has not yet been
possible due to time constraints. Problems noted with the analyses to
date will be investigated further. Aircraft data (including flight logs)
will be re-examined for better indications of cloud top height.
Different spatial averaging schemes will be tested to improve the
representation of GDCP values in the aircraft observing region.
Continued work on the case studies presented here and on other cases we
may identify will provide more insight.

In addition to a more extensive analysis of the products already
considered, future work will involve comparison of other variables
available from GDCP. Liquid water path and ice water path will be
calculated from aircraft data when possible to be compared with GDCP
estimates of LWP/IWP. Similarly, effective radius/diameter will be
derived from in situ observations of cloud droplet spectra for
comparison with the GDCP Reff/Deff product. The LWP product will also be
compared with ground-based microwave radiometer LWP measurements when
they are available.

**SUMMARY**

Although encouraging, the methods and results presented here should be
considered preliminary. CIP already does an excellent job at diagnosing
the potential for icing on a 20-km scale, but we hope to do better. Our
plans for this product include the depiction of icing severity, better
information on locations of supercooled large droplets (SLD), and higher
resolution products for the terminal area. We have only modest skill to
date on severity and SLD and hope that the advanced satellite products
will help increase our skill. For the terminal area, resolution of 5 km,
or smaller, and time scales of 15-30 min or shorter are needed, and
these cannot be accomplished without satellite support. We already know,
in a qualitative manner, the value of the advanced satellite products
through our experience using them in field project forecasting exercises
and it is now time to quantify their accuracy.

We know have available a number of unique sets of observations that
should enable us to determine which advanced satellite products are the
best for incorporation into CIP. We seek increased efficiency (greatest
icing detection with smallest overwarning) and an improved icing
severity algorithm. We also expect that the advanced satellite cloud
products will enable more accurate icing diagnosis in areas where
surface data are sparse, for example, over oceans and Alaska.

**REFERENCES**

Bernstein, B.C., F. McDonough, M.K. Politovich and B.G. Brown, 2004:
CIP: physically-based, integrated approach to the diagnosis of in-flight
aircraft icing, Part I: Algorithm description. Submitted, *Weather and
Forecasting (was submitted last year under a different name to a
different journal)*

Brown, B.G., 1996: Verification of in-flight icing forecasts: Methods
and issues. *Proceedings, FAA International Conference on Aircraft
In-flight Icing,* Report No. DOT/FAA/AR-96/81, II, 319-330.

Brown, B.G., G. Thompson, R.T. Bruintjes, R. Bullock, and T. Kane, 1997:
Intercomparison of in-flight icing algorithms. Part II: Statistical
Verification results. Weather and Forecasting, 12, 890-914.

Brown, B.G., T.L. Kane, R. Bullock, and M.K. Politovich, 1999: Evidence
of improvements in the quality of in-flight icing algorithms. Preprints,
8th Conference on Aviation, Range, and Aerospace Meteorology, Dallas,
TX, 10-15 January, American Meteorological Society (Boston), 48-52.

Minnis, P., D. P. Garber, D. F. Young, R. F. Arduini, and Y. Takano,
1998: Parameterization of reflectance and effective emittance for
satellite remote sensing of cloud properties. *J. Atmos. Sci.*, **55**,
3313-3339.

Minnis, P., D. P. Kratz, J. A. Coakley, Jr., M. D. King, D. Garber, P.
Heck, S. Mayor, D. F. Young, and R. Arduini, 1995: Cloud Optical
Property Retrieval (Subsystem 4.3). \"Clouds and the Earth\'s Radiant
Energy System (CERES) Algorithm Theoretical Basis Document, Volume III:
Cloud Analyses and Radiance Inversions (Subsystem 4)\", *NASA RP 1376
Vol. 3*, edited by CERES Science Team, pp. 135-176.

Minnis, P., W. L. Smith, Jr., L. Nguyen, J. J. Murray, P. W. Heck, and
M. M. Khaiyer, 2003: Near-real-time satellite cloud products for icing
detection and aviation weather over the USA. *Proc. FAA In-Flight
Icing/De-icing International Conference*, Chicago, IL, June 16-20,
CD_ROM, 2003-01-2097.

Minnis, P., W. L. Smith, Jr., L. Nguyen, M. M. Khaiyer, D. A.
Spangenberg, P. W. Heck, R. Palikonda, B. C. Bernstein, and F.
McDonough, 2004: A real-time satellite-based icing detection system.
*Proc. 14^th^ Intl. Conf Clouds and Precip.- ICCP 2004*, Bologna, Italy,
July 18-23.

Minnis, P., W. L. Smith, Jr., D. F. Young, L. Nguyen, A. D. Rapp, P. W.
Heck, S. Sun-Mack, Q. Trepte, and Y. Chen, 2001: A near-real time method
for deriving cloud and radiation properties from satellites for weather
and climate studies. *Proc. AMS 11th Conf. Satellite Meteorology and
Oceanography*, Madison, WI, Oct. 15-18, 477-480.

McDonough, F. and B.C. Bernstein, 1999: Combining satellite, radar and
surface observations with model data to create a better aircraft icing
diagnosis. *Proceedings, 8th Conference on Aviation, Range and Aerospace
Meteorology*, Dallas, 10-15 January. Amer. Meteor. Soc., Boston,
467-471.

Smith, W. L., Jr., P. Minnis, B. C. Bernstein, F. McDonough, and M. M.
Khaiyer, 2003: Comparison of supercooled liquid water cloud properties
derived from satellite and aircraft measurements. *Proc. FAA In-Flight
Icing/De-icing International Conference*, Chicago, IL, June 16-20,
CD_ROM, 2003-01-2156.

Smith, W. L., Jr., P. Minnis, B. C. Bernstein, A. D. Rapp, and P. W.
Heck, 2002: Supercooled liquid water cloud properties derived from GOES:
Comparisons with in situ aircraft measurements. *10^th^ AMS Conf.
Aviation, Range, and Aerospace Meteorol.*, Portland, OR, May 13-16,
89-92.

Smith, W. L., Jr., P. Minnis, and D. F. Young, 2000: An icing product
derived from operational satellite data. *Proc. AMS 9^th^ Conf.
Aviation, Range, and AerospaceMeteorol.*, Orlando, FL, 11-15 Sept.,
256-259.

**Appendix: Tasking for Benchmarking Exercise**

**Benchmarking CIP and NASA Langley Satellite Products**

**For Future Upgrades**

Joint Project of NCAR, CIMMS and NASA LaRC

Revised Outline 040416

[Goal]{.underline}: To assess current capabilities of the Current Icing
Potential (CIP) and NASA LaRC GOES-Derived Cloud Products (GDCP) in
diagnosing inflight aircraft icing conditions

[Specific questions to be addressed]{.underline}

-   What are the current PODs (y/n), areas and volumes covered, and
    efficiencies of the products?

-   How well do the products diagnose expected icing severity?

-   What weather scenarios do the products handle well? Poorly?

-   Where could GDCP provide greatest benefit through incorporation into
    CIP?

[Task 1:]{.underline} PIREP-based verification of CIP and Satellite
Products

[Task 2:]{.underline} Anecdotal verification of selected cases using
research aircraft data

[Task 3:]{.underline} Preliminary examples of linked products

**Task 1**

Icing PIREPs will be used to verify the accuracy of CIP and satellite
icing products using methods already established by the IFIPDT.

Verification period: October 2003 through March 2004

Verification time: 1500 and 2100 UTC

Statistics: PODy, PODn, area covered, volume covered, efficiency
Verification volume: CONUS from surface to 18,000 ft

Products: CIP, CIP severity (MOG), NASA Severity, Liquid water path,
cloud phase, effective radius, cloud top and base heights, NCAR
satellite icing product

**Task 2**

Recent field research program data will be used to examine selected
cases and provide insight on the behavior of the products.

Projects: AIRS-2, THORPEX, WISP04, NASA GRC icing missions

Products: CIP, CIP severity (MOG), selected GDCPs, RAP satellite product

Data sets: Research aircraft (liquid water content, cloud phase, icing
rate), ground-based radiometer (liquid water path), \[we discussed radar
data as well\]

Study days: 10-11 March (WISP04), 24 November (AIRS-2), 27 November
(AIRS-2), others TBD \[Pat and I have also agreed upon 30 November
(AIRS-2/THORPEX)\]

**Task 3**

CIP output will be combined with one or more NASA satellite products to
simulate an integrated product and predict benefits.

Based on the results of Tasks 1 and 2, CIP output fields will be
modified by NASA Satellite output fields to produce an enhanced version
of CIP. For this task only a few cases will be done with very
rudimentary combining techniques to demonstrate potential benefits.

Example days: Chosen from Task 1 and Task 2 analyses

#### 

[^1]: Note that CIP outputs altitudes in ft rather than m, since these
    are the units used by the aviation end-users. We will retain these
    units in this report rather than converting to metric.