**AnalogPoP**

**version 1.1**

Les Colin -- WFO Boise, ID

AnalogPoP is similar to the Analog tool for temperatures as it improves
on GFE model PoP forecasts by adjusting to the errors the models made in
similar PoP situations in the past. It does this using archives of past
model forecasts and observed PoP grids stored in the BOIVerify (Barker,
2006) database.

However, forecasting and verifying PoPs are very different from
forecasting temperatures and these differences make the AnalogPoP tool
quite different from the Analog tool.

Unlike temperature, PoP is limited to a range of zero to 100. Also, PoPs
in shorter time periods do not combine for longer time periods the same
way temperatures do. The Max temp at a point over a given time period is
simply the highest temperature at that point in all the shorter time
periods contained within the larger time period. "Floating PoP" operates
the same way, but the true PoP behaves differently. For example, if
00z-06z PoP is 40%, and 06z-12z PoP is 80%, then the 00z-12z floating
PoP is 80% (the larger of the two values). But the true 00z-12z PoP is
88%. This value is based on the [complements]{.underline} of the given
PoPs: In the 00z-06z period the probability of no rain (i.e., the
complement) is 60% (100-40). The probability of no rain in the 06z-12z
period is 20% (100-80). The probability of no rain in the 00z-12z period
is the product of these complements, or .60x.20, which is .12, or 12%.
The remainder, 88%, is the probability of rain in either or both 6-hour
periods, and this value is the true 12-hour PoP. For mathematical
consistency the AnalogPoP tool only uses the true PoPs.

For verification of temperature the Analog tool uses a single scoring
metric to compare the accuracy of various model and human forecasts.
This metric is a simple function of mean absolute error. The smaller the
error, the better the forecast.

Similarly, we need a single scoring metric for PoP in order to compare
the accuracy of model and human forecasts. However, there is no physical
reality for PoPs between zero and 100; it either rains or it doesn't.
Physical reality belongs only to PoPs of either zero or 100. Determining
a single comprehensive scoring metric---i.e., some value that describes
the overall quality of a PoP forecast \--is not easy, and the
alternative of using more than one metric would make comparisons too
complicated. An additional consideration is that it is easy to forecast
zero PoP on a clear day and earn the maximum possible verification
score. It's almost as easy to forecast 100 PoP when it rains everywhere.
We don't want these scores to rise to the top of the verification list.

It's harder to correctly forecast PoP where some areas get rain and some
don't. The most difficult forecast is probably the one where rain will
fall randomly over 50 percent of a region. Certainly a forecast which
identifies the rain areas exactly right should receive more credit than
the zero or 100 PoP examples.

With these ideas in mind we develop our PoP scoring metric as follows:

At a given grid box (i.e., point) A, we compute a "Brier value" for the
complement of the PoP. For example, if PoP(A) is 30, then the complement
is 70 and the Brier value is 49.

Next, we search the nearest neighbors of A (actually we search a square,
not a circle, centered at A). If the observed pcpn at a given
neighboring grid box B is opposite that of A (that is, if it rains at A
but not at B, or vice versa), then we compute the PoP difference between
A and B. In effect, we cross the rain/no rain boundary between A and B.
(Actually, we cross the boundary an odd number of times between A and B,
but the tool assumes exactly one crossing. If the boundary is crossed an
even number of times it is ignored. For example, dry at A, wet somewhere
between A and B, then dry again at B, means dry at both A and B and
therefore ignored). Now we compute a contribution from B as the value
\[PoP(B)-PoP(A)\]\*\*2/(dist(A-B))\*\*2, provided PoP(B) differs from
PoP(A) in the right sense. By "right sense" we mean that PoP(B) is
greater than PoP(A) if it rains at B but not at A. If PoP(B) differs
from PoP(A) in the wrong sense, the contribution is subtracted as a
penalty. The contribution from each point B in the edit area is added to
A's Brier value. A higher score therefore represents a better forecast.

This calculation is made for each point A in the forecast region. The
average of the values among all points A is the overall score. Finally,
the scores are divided by 10 as a scaling operation. Now a zero PoP on a
clear day gets a score of 10, not 100. A flat 100 PoP on a day where it
rains everywhere also gets 10. A flat 50 PoP gets 7.5, not 75.

Some further explanation about the search area is needed here: Instead
of searching the nearest 100 neighbors around a given point A, we thin
the search in proportion to the square of the distance from A. That is,
fewer neighbors are searched at greater distances from A. This allows us
to search a greater distance from A for the rain/no rain boundary (which
is assumed to be crossed only once between A and B), otherwise with 2.5
km grid boxes we can only search about 12.5 km around each point A. Note
that thinning the search is not the same thing as searching every
neighbor and then dividing their contributions by their squared distance
from A.

Now that we have a scoring metric how can we use it? First, even though
the score is open-ended (i.e., can exceed 100) we can limit it so that
scores above 100 become 100. Second, since a uniform zero or 100 PoP
forecast never earns a bonus (since at every point PoP(B)==PoP(A)), the
highest possible score for those cases is only 10.

This scoring system encourages a forecaster to be more definite in
delineating the rain/no rain boundary. Only then can bonuses be earned.
But when the Brier value for A is low there are usually fewer points B
with PoPs that differ in the right sense from PoP(A) across the rain/no
rain boundary, and bonuses are harder to earn.

But the main thing is that we now can see whether analogs improve the
PoP forecast or not. We can also compare how various models score
against each other and against us.

**[How AnalogPoP Works:]{.underline}**

For a given model PoP grid, AnalogPoP looks into the BOIVerify
historical database for PoP grids made by the same model and forecast
period. For the archived grids found to be most like the current PoP
grid, the tool retrieves the 12-hour precipitation (QPE) grids that
occurred on their verification dates, and creates a new grid from them.
For example, if it rained on 8 of the 25 verification dates at a
particular grid box, the created PoP for that grid box would be
approximately 32. The reason it would not be exactly 32 is that we also
weight the 8 occurrences by how similar their overall PoP grids were to
the current PoP grid on the default CWA. The calculation is repeated for
every grid box on the screen, even on grid boxes outside the default
CWA. The created grid is now used instead of the weighted average PoP
grid for the analog dates. Now, since each analog PoP grid probably
differed from the current PoP grid, their average difference from the
current PoP grid must be added back on as a final step. For example, if
the 25 best PoP analogs averaged 10 percent drier than the current PoP
grid, we must add 10 percent to the created grid as the final step.

**[Tool Features:]{.underline}**

The AnalogPoP tool GUI is shown below:

![](media/image1.png){width="3.7263888888888888in"
height="6.5472222222222225in"}

Starting in the upper left of the GUI you can choose either PoP0012 or
PoP1200. These are radio buttons; choosing one turns off the other. PoP
is defined only on 12 hour periods starting at 00z or 12z.

In the top center and top right of the GUI you can choose as many models
as you want. These are check-buttons. The model solutions will be
blended together at the end.

Next is a differential smoother. The tool smoothes the grid using all
points within the specified elevation range, and outward as many grid
boxes as specified in the smoothing radius. The default smoothes each
grid box by all boxes within 3 boxes of a given box, provided they are
within 2000 feet elevation of the given box. If you set both values to
zero you get an unsmoothed PoP formed from the number of days each grid
box received pcpn divided by the total number of days used. This output
will look choppy if the number of days used is small. For example, if
the number of days used is ten, then PoP resolution can be no finer than
every ten percent. The smoother irons out this choppiness.

Next, decide how many past days you want to examine and how many of them
to use. Here you choose to examine the past 100 days. The 100 analogs
are ranked best to worst and you want to use only the best 25 of them.
Make sure that "Days examined" is larger than "Days Used". The power of
the analog approach is that it is selective, i.e., finds the best
subset, of those past days.

The ranking process for PoP similarity is not quite the same as for
temperatures. With PoP it is only necessary to compare the current and
past PoPs at each grid point. (With temperatures the first step is to
subtract out the mean temperature of each grid leaving only the
residuals \[departures from the means\], and then compare the residual
fields to the residual field of the current temperature grid. This step
is necessary to account for seasonal differences.) The differences are
squared (to emphasize the larger differences) and then summed over all
points in the CWA domain, resulting in a single large number. The
smaller this number the better is the overall pattern match. The past
dates are ranked in ascending order of these numbers and the analogs are
chosen from the top of the ranked list.

Next is: "With more than one model selected use common analog-dates
only?" By saying "Yes" you only use those dates found for both the GFS40
and NAM12. The models taken separately may identify different dates for
their 25 best analogs. Are any of those dates the same in both lists? If
"Yes", use only those analogs, otherwise use all the analogs in both
lists.

Next is a series of slider bars. The forecast period is determined from
launch time of the forecast. For example, if you are on Day shift, then
forecast period 1 is the following 00z-12z timeframe, forecast period 2
is the 12z-00z period after that, and so on. If you are on Mid shift,
forecast period 1 is 12z-00z today, forecast period 2 is 00z-12z after
that, and so on.

You must also specify the target date of the forecast. If this is day
shift on Sep 5, then 5^th^ period PoP0012 will be for 00z-12z Sep 8. It
may seem redundant to specify both the forecast period [and]{.underline}
the target date, but it is necessary when working on verification of
past dates.

Finally, at the bottom of the GUI are four more sections which are
available only when a single model is selected. These are: "Details?",
"Recalculate?", "Restore?", and "Verify custom grid?".

The first three are available only for future dates. "Verify custom
grid?" is available only for past dates. These will be described in
turn.

The first one is "Details?" If you choose "Yes", you get a lot of
information about all the analogs involved in the calculations:

![](media/image2.png){width="5.9944444444444445in"
height="2.1173611111111112in"}

The upper-most grid, entitled "G40", is the final analog-adjusted PoP,
the same grid you get if you set "Details?" to "No". The next line
(G40Analog) is a series of 25 short-duration grids (we assume 25 "days
used" in this example) and one longer-duration grid. The longer grid
contains the original unmodified GFS40 PoP. The 25 shorter grids are the
analog PoPs, with the rightmost one the best match to the original GFS40
PoP, the next one to the left the second-best match, and so on. These
grids correspond, in order, to the dates shown in the terminal window.
By stepping leftward through the analogs you can see how well they
matched the original GFS40 PoP.

The next line ("G40AnalogDiffs") has the difference grids from the
original GFS40. These are shown as short-duration grids for each of the
25 analogs. The longer-duration grid on the right has the weighted
average of the 25 difference grids. The weighting is in proportion to
how close the analog PoPs were to the original GFS40 PoP, with the most
similar one receiving the greatest weight. By adding the longer grid in
this line to the original GFS40 PoP in the line above it, you get the
weighted PoP of all the analogs. That grid can be seen when the
"Recalculate?" button is set to "Yes" (described later).

The next two lines ("G40 QPE0" and "G40 QPE1") have the 25 6-hour QPE
observed precipitation grids corresponding to the analog dates. These
are shown as short-duration grids. For each analog date, QPE0 has the
first 6 hours (12z-18z in this example) and QPE1 has the second 6 hours
(18z-00z). The longer-duration grid to the right holds the averages of
the 25 QPE0 (or QPE1) grids. When "Recalculate?" is set to "Yes", 6-hour
PoPs are computed at every grid point as a ratio of the number of times
precipitation occurred, to the number of "days used" (25 in this
example). So if rain fell at BOI on 8 of the QPE0 grids, the 12z-18z PoP
at BOI would be 32%, and if rain fell at BOI on 5 of the QPE1 grids the
18z-00z PoP at BOI would be 20%. These two computed 6-hour PoP grids are
shown as short-duration grids to the right of the G40 grid, and are
followed by the true 12-hour PoP formed from the complements of the two
6-hour PoPs. If you have not removed or changed any of the QPE0 or QPE1
grids, this 12-hour PoP grid minus the weighted average difference grid
will exactly match the unsmoothed G40 grid. (You can see this by zeroing
the differential smoother beforehand.)

But "Recalculate?" serves another purpose: If you delete or change any
of the shorter-duration grids "Recalculate?" will update all the
averages, i.e., longer-duration grids. For example, suppose you decide
not to use one of the GFS40 analogs. You can delete that analog along
with the corresponding GFSAnalogDiffs, QPE0, and QPE1 entries. Now if
you check "Yes" on "Recalculate?" and rerun the tool, new calculations
will be made using only 24 days. You can delete or change as many
analogs as you want and then use "Recalculate?".

Next is the "Restore?" button. If you have deleted or changed any of the
Analog entries, checking "Yes" here and re-running the tool will restore
all the original Analogs and averages, and you can start over. This is
much faster than re-running the tool using the "Details?" option.

Finally, there is a button called "Verify custom grid?" This will be
explained in the verification section below.

**[PoP Verification:]{.underline}**

The value of AnalogPoP can only be seen through verification, and
verification, of course, can only be done on past dates.

When you run AnalogPoP on a past date you get an error-frequency
histogram, some numerical information in the terminal window, and (if
you are verifying only one model) five new grids.

The five new grids when verifying only one model are:

1\. Custom grid---initially a zero-grid.

2\. F grid---the forecast grid corresponding to the GUI choices for
model, days examined, and days used.

3\. O grid---the observed PoP grid, containing only 100 where measurable
precipitation was observed, and 0 where precipitation was not observed.

4\. OF grid---a grid of F minus O.

5\. SignedBrier---positive where F\>0 and O is zero, and negative where
F\<100 and O is 100.

The histograms are usually bi-modal because where rain fell the forecast
PoP was usually less than 100, i.e., too small, and where rain did not
fall the forecast PoP was usually greater than zero, i.e., too large. A
perfect forecast would have 100 PoP wherever rain fell and zero PoP
everywhere else. Here is a sample histogram:

![](media/image3.png){width="5.9944444444444445in"
height="4.978472222222222in"}

In this histogram one of the modes is near -50 on the x-axis and the
other is near +10. At the top of the histogram are the number of points
verified (22199 in BOI's entire CWA). Total Score (7.97) is the sum of
Raw Score (7.45, which is the Brier value) and Bonus (0.52, as explained
in the introduction). Next the Model is identified---N12 represents
NAM12. N is the number of models blended (just 1 here, N12).
WetPoPError, -49.14%, is the average PoP deficit from 100% for the 19332
wet points. DryPoPError, 12.15%, is the average PoP excess above 0% for
the 2867 dry points.

The graph of the histogram shows the frequency of these PoP errors.
Since, as noted earlier, the average PoP deficit for the wet areas was
-49.14%, the average PoP there was 50.86%. A better forecast would have
had higher average PoP for the wet areas. The average PoP excess for the
dry areas was 12.15%. Black arrows along the x-axis mark these values.
The red arrow indicates the average PoP error (-41.22%) for all 22199
points.

The terminal window reports information about the analogs themselves.
Suppose you are verifying the GUI settings below:

![](media/image4.png){width="2.876388888888889in" height="5.675in"}

The terminal window displays the following:

![](media/image5.png){width="5.998611111111111in"
height="2.5166666666666666in"}

In the GUI we requested 3 "days used" from 10 "days examined", and the
terminal window reports information for the top 3 analogs for both the
GFS40 and NAM12. Reading from left to right are the year, month, date,
epoch verifying-start-time (seconds since Jan 1 1970), epoch
verifying-end-time, similarity score (lower is better), and the model
number archived in BOIVerify. The analogs for each model are sorted by
similarity score, smallest at the top.

"nn= 3" means that 3 grids were actually found and used for each model.
"Score" is the total score (7.46), which is the sum of the Brier value
(7.07) and bonus (0.39).

In the "Tool features" section we mentioned the button "Verify custom
grid?". If you were verifying only one model (say NAM12 alone) you got
five new grids. One of those grids was a "Custom" grid, initialized to
zero. You can copy any PoP grid into that slot and verify it by checking
"Yes" to "Verify custom grid?". Each verification produces a new
histogram and new scores. When you are verifying the custom grid all the
other GUI settings are disabled.

Some examples:

1\. Copy the "O" grid into "Custom". The "Custom" grid is now a perfect
forecast, i.e., it has 100 PoP wherever it did rain, and 0 PoP where it
didn't. It also means that the "Custom" grid identifies the rain/no rain
boundary perfectly, so it should get a huge bonus.

Here is the histogram:

![](media/image6.png){width="5.9944444444444445in"
height="4.978472222222222in"}

Notice the bonus score: 196.28. The Raw score is 10.00, the maximum
possible, and WetPoPError and DryPoPError are both 0.00. The red arrow
at 0.00 is right on top of the two black arrows so the black arrows
cannot be seen. The y-axis goes past 22000 to show that all 22199 points
verified with zero error.

The total score of 206.28 is the highest score possible for this case,
but other cases could have even higher scores. Here is the O (or Custom)
grid for this case:

![](media/image7.png){width="3.4340277777777777in" height="4.75in"}

Because most of the CWA (hatched area) had rain, the rain/no rain
boundary was relatively short. But a longer or more irregular boundary
would have earned a larger bonus.

2\. Same as example 1, but applying the differential smoother within
2000 feet and 3 grid boxes of every grid box in the CWA. The Custom grid
shows how the smoother affected the rain/no rain boundary:

![](media/image8.png){width="3.515277777777778in"
height="4.690277777777778in"}

Here's the resulting histogram:

![](media/image9.png){width="5.9944444444444445in"
height="4.978472222222222in"}

Note that the scores have been lowered by the smoother.

3\. Now let's verify the original NAM12 alone for the same date, i.e.,
using no analogs, and with the smoother turned off (0 and 0). Here's the
histogram:

![](media/image10.png){width="5.9944444444444445in"
height="4.978472222222222in"}

Here is the original NAM12:

![](media/image11.png){width="3.50625in" height="4.6409722222222225in"}

4\. Now the same N12 as in example 3 but using one analog in the
previous 10 days (days examined=10, days used=1), and smoother still
turned off.

The histogram looks like this:

![](media/image12.png){width="5.9944444444444445in"
height="4.978472222222222in"}

The histogram score has improved (from 6.82 to 8.93) largely because we
approximated the rain/no rain boundary better than the original NAM12
did. But because we used QPE from only one analog date to Sep 20, i.e.,
QPE for Sep 19, the computed PoP grid looks terrible, as shown below:

![](media/image13.png){width="3.50625in" height="4.6409722222222225in"}

Using the differential smoother with 2000 feet and smoothing radius 3
improves the appearance of the preceding image:

![](media/image14.png){width="3.50625in" height="4.6409722222222225in"}

...but lowers the histogram score from 8.93 to 8.29:

![](media/image15.png){width="5.9944444444444445in"
height="4.978472222222222in"}

5\. November 3, 2008 was a rainy day on about half the BOI CWA, as the
observed 12-hour PoP (O grid) shows:

![](media/image16.png){width="3.50625in" height="4.6409722222222225in"}

The original 1^st^ period GFS40 verifying on 12z-00z Nov 3 is shown
below:

![](media/image17.png){width="3.50625in" height="4.6409722222222225in"}

\...and it produced this verification histogram:

![](media/image18.png){width="5.9944444444444445in"
height="4.978472222222222in"}

Note that the bonus score here is nearly zero, indicating no skill in
identifying the rain/no rain boundary.

AnalogPoP on the same 1^st^ period GFS40 for 12z-00z Nov 3 with 70 days
examined and 10 days used, and default smoother, produced this
analog-adjusted GFS40 forecast:

![](media/image19.png){width="3.50625in" height="4.6409722222222225in"}

...and this verification histogram:

![](media/image20.png){width="5.9944444444444445in"
height="4.978472222222222in"}

We see that the total verification score improved from 7.04 to 8.86.
Average PoP in the wet areas increased from 31.89% (the wet deficit was
68.11%) to 42.82%, but the average PoP in the dry areas also increased
from 29.78% to 40.00%. The bonus score increased from -.03 to +1.53.

From the terminal window listing the best 1^st^ period GFS40 PoP analog
verifying on 12z-00z November 3 verified on October 3 (i.e., was made on
the 00z cycle October 3):

![](media/image21.png){width="5.996527777777778in"
height="3.092361111111111in"}

Here is that GFS PoP (verifying Oct 3, 2008) (left) alongside the
original GFS40 (verifying Nov 3, 2008) for comparison (the comparison
being made only within the CWA):

![](media/image22.png){width="2.6770833333333335in"
height="3.542361111111111in"}
![](media/image23.png){width="2.6770833333333335in"
height="3.542361111111111in"}

The GFS40 image on the left is the very best match to the GFS40 image on
the right in the 70 days prior to November 3. It is by no means a
perfect match.

**[Advice on Using this Tool:]{.underline}**

It's best to start GFE from a terminal window rather than from a menu.
Just left-click in an empty area on the AWIPS screen and select
"Terminal". When the terminal window opens type:

runGFE, and hit enter.

The reason for doing this is that AnalogPoP will output some of its
information to the terminal window where you can see it.

Don't use AnalogPoP without considering in advance what the analogs
might be. For example, suppose the current GFS40 predicts rain for a
large part of your CWA for tomorrow, but it has only rained once in the
past 100 days. If you choose 100 for "days examined" and 20 for "days
used", you will get back [at most]{.underline} only the one analog QPE
that had rain. But since you asked for 20 "days used", the largest
possible PoP anywhere in the CWA can only be 5% (if you had asked for 10
"days used" the largest possible PoP could only be 10%). This is almost
certainly not what you want. What should you do now?

You can either re-run AnalogPoP with fewer days used, or re-run
AnalogPoP with "Details?" checked "Yes", then delete the poorer analogs,
followed by a "Recalculate?".

The above example is meant to show that AnalogPoP works best during the
middle of the rainy season or dry season, where there are usually enough
good past examples to compare. During transition periods, e.g., the
first rainy day after a long drought, or the first dry day after many
wet days, AnalogPoP will not work as well.

A different problem relates to PoP resolution. If you set "days used" to
10, say, then the computed PoPs throughout the CWA can only be in
increments of 10 (not counting the smoother), because there can only be
0,1,2,3,...,10 QPE days among the 10 "days used", so the computed PoPs
at any point on the screen can only be 0%,10%,20%,30%,...,100%. The
smoother will homogenize the values but you should realize that this is
happening.

AnalogPoP allows you to examine up to 200 past days and use up to 50 of
them. This can take several minutes for even one model. A blend of
several models takes even longer. If you need to stop AnalogPoP while it
is running, hit "control-c" in the terminal window. But only do this
when the tool is running---if you do it when no tool is running you'll
terminate GFE.

You should realize that the PoP grids archived in BOIVerify are in
"floating PoP" form for some, but not all, of the models. For the
ADJMAV, ADJMET, and MOSGuide, the archived PoPs are just as they appear
in AWIPS. For NAM12 a different process is used:

NAM12 outputs QPF every 3 hours and a PoP is created four times from the
QPF every 12 hours. The 12 hour NAM PoP is the floating (i.e., largest)
PoP of the four grids at every grid point. The GFS40 outputs QPE every 6
hours, so the 12 hour GFS40 PoP is the floating PoP of two 6-hour
created-PoPs. With SREF the archived 12-hour PoP at any grid point is
the ratio of the number of ensemble members which output QPF\>=.01" to
the total number of ensemble members. If 7 members (out of 21 total)
output measurable precipitation at a given grid box, the PoP at that
grid box would be 33%.

Sometimes AnalogPoP outputs small scale maxima, or localized high PoPs.
You can (and should) remove them using the pencil tool.

Finally, if you have used the Analog tool (for temperatures) before
using the AnalogPoP tool you may find you cannot copy and paste among
certain grids, particularly into the "Custom" slot. This is because the
"O", "OF", and named model-grids (like "G40", "N12MET", etc) still have
display settings for temperature rather than PoP. Be sure to unload all
temperature grids created by Analog before running AnalogPoP.

**Tool Installation:**

You must have BOIVerify already loaded to use this tool. BOIVerify must
also be archiving PoP and QPE grids at your site.

Use the ifpServerText program to put the Analog tool into user SITE by
entering the following command:

ifpServerText --u SITE --s --n AnalogPoP --f AnalogPoP.tool --c
SmartTool

Near the top of the execute section change the default CWA to the name
of the edit area that represents your entire CWA.

If you still have problems e-mail me (<les.colin@noaa.gov>) and I'll try
to help you.