# Evaluating the RELM Test Results

Authors: Michael K. Sachs; Ya-Ting Lee; Donald L. Turcotte; James R. Holliday; John B. Rundle
Journal: International Journal of Geophysics
 (2012)

---

## Abstract

We consider implications of the Regional Earthquake Likelihood Models (RELM) test results with regard to earthquake forecasting. Prospective forecasts were solicited forM≥4.95 earthquakes in California during the period 2006–2010. During this period 31 earthquakes occurred in the test region with M≥4.95. We consider five forecasts that were submitted for the test. We compare the forecasts utilizing forecast verification methodology developed in the atmospheric sciences, specifically for tornadoes. We utilize a “skill score” based on the forecast scores λfi of occurrence of the test earthquakes. A perfect forecast would have λfi=1, and a random (no skill) forecast would have λfi=2.86×10-3. The best forecasts (largest value of λfi) for the 31 earthquakes had values of λfi=1.24×10-1 to λfi=5.49×10-3. The best mean forecast for all earthquakes was λ̅f=2.84×10-2. The best forecasts are about an order of magnitude better than random forecasts. We discuss the earthquakes, the forecasts, and alternative methods of evaluation of  the performance of RELM forecasts. We also discuss the relative merits of alarm-based versus probability-based forecasts.


---

## Body

## 1. Introduction

Earthquakes do not occur randomly in space. Large earthquakes occur preferentially in regions where small earthquakes occur. Earthquakes are complex phenomena, but they do obey several scaling laws. One example is Gutenberg-Richter frequency-magnitude scaling. The cumulative number of earthquakesN with magnitudes greater than M in a region over a specified period of time is well approximated by the relation(1)log⁡N=a-bM,
where b is a near universal constant in the range 0.8<b<1.1 and a is a measure of the level of seismicity. Small earthquakes can be used to determine a and (1) can be used to determine the probability of occurrence of large earthquakes. Kossobokov et al. [1] utilized the number of M≥4 earthquakes in 1°×1°areas to map the global seismic hazard.A question that has been studied by many groups is whether there are temporal variations in seismicity that can be used to forecast the occurrence of future earthquakes. Earthquakes on major faults (say the San Andreas in California) occur quasiperiodically. A reasonable hypothesis would be that the rate of regional seismicity would accelerate during the period between the major earthquakes. There is no evidence that this occurs systematically. Background seismicity in California appears to be stationary. With the exception of years with large aftershock sequences, Rundle et al. [2] (Figure 1) showed that seismic activity in Southern California in the magnitude range 1.5<m<4 for the period 1983 to 2000 was well represented on a yearly basis by (1) taking a=5.4 and b=1.0.Figure 1
Map of the test region, the coast of California, major faults, and the 31 earthquakes withM ≥ 4.95 that occurred in the test region. The earthquakes are given in Table 1.Intermediate-term earthquake forecasting algorithms based on pattern recognition of variations in regional seismicity were developed by Keilis-Borok and colleagues [3]. These forecasts were alarm based, when a threshold of anomalous behavior was reached a warning of a time of increasing probability (TIP) of an earthquake was issued. A relatively high success rate was found including the 1988 Armenian earthquake and the 1989 Loma Prieta earthquake [4], but there were also notable false alarms and failures to predict.The focus of this paper is to study the implications of the RELM test of earthquake forecasts in California. This was a prospective test of forecasts form>5 earthquakes during the period 2006–2010. Forecast submission was required prior to the starting date. In our study of the RELM test results we will utilize the methodology developed in the atmospheric sciences [5], specifically for tornadoes. Tornado forecasts are alarm based. Two levels of alarms are issued: (1) a tornado watch is issued for a specified area and time if atmospheric conditions appear conducive to tornados, (2) a tornado warning is issued if one or more tornados have been observed. The evaluation of tornado forecasts is based on the number of failures to predict and on the number of false alarms. A quantitative measure of success is the skill score, the skill score is unity for a perfect forecast and zero for a random (no skill) forecast. RELM forecasts were probabilistic rather than alarm based, that is a continuous range forecast probabilities were required. In an alarm-based forecast an area of high risk is specified. We will discuss the implications of the two alternative approaches.The forecasts submitted to the RELM test were primarily based on precursory seismic activity. There are a variety of approaches to the quantification of this activity. In Section2 of this paper we will discuss the relative intensity (RI) and pattern informatics (PI) approaches. The RI approach extrapolates the occurrence of small earthquakes during a specified precursory time window. High activity (activation) indicates high risk. The PI approach is related but includes both activation and quiescence. In Section 3 the problems with retrospective forecasts are discussed. In Section 4 the RELM test is discussed and the test earthquakes are described in Section 5. The submitted forecasts are discussed in Section 6 and are evaluated in Section 7.An objective of this paper is to understand the relationship of the forecasts to the distribution of seismicity during the test period. We discuss what we believe is a well-defined precursory activation.


## 2. PI and RI

A pattern informatics (PI) approach to earthquake forecasting was proposed by Rundle et al. [2, 6] and Tiampo et al. [7]. In forecasting M≥5 earthquakes a region is divided into a grid of 0.1°×0.1°regions. The rates of seismicity in the regions are studied to quantify anomalous behavior. Precursory changes that include either increases or decreases in seismicity are identified during a prescribed time interval. If changes exceed a prescribed threshold hot-spots are defined. The forecast is that future M≥5 earthquakes will occur in the hot spot regions in a 10-year time window. Thus, the PI method is alarm based. Utilizing the PI method Rundle et al. [8] made a forecast of California hot spots valid for the period 2000–2010. Holliday et al. [9] reported that 16 of the 18 earthquakes that occurred during the period 2000–2005 occurred in hot spot regions. The PI forecast is time dependent because it is based on temporal changes in background seismicity.A closely related forecasting technique is the relative intensity (RI) approach. The RI forecast is based on the direct extrapolation of the rate of occurrence of small earthquakes using (1). The RI forecast can be time dependent if the time span of the background seismicity is relatively short. The success of the PI method described above led to a discussion as to whether the PI method is significantly better than the RI method. Comparisons of these approaches have come to different conclusions regarding their validity [10, 11]. These comparisons emphasize the difficulties in evaluating the performance of seismicity forecasts.


## 3. Prospective versus Retrospective Forecasts

A prospective forecast is a true forecast of future earthquakes. No knowledge of these earthquakes exists. A retrospective forecast is a forecast of earthquakes that have occurred in the past (say 2000–2010) based on data available before the start of the period. The existence of the forecast earthquake is known. In principal a retrospective forecast can be carried out fairly; however, in many cases these forecasts are biased by the existence of the forecast earthquakes.The PI forecast by Rundle et al. [8] was prospective. However, the successful forecast of 16 out of 18 earthquakes in California led to a retrospective challenge of the results [11].It became clear that it would be desirable to sponsor a contest in which research groups would provide prospective forecasts of earthquakes under well-defined conditions. This was the origin of the RELM test, which will be described in the next section. Some of the rules were based on the prospective forecast made by Rundle et al. [8]. The test region was California. Forecasts were made for M>5 earthquakes on a grid of 0.1°×0.1°forecast cells. The forecast period was 1 January 2006 to 31 December 2010. The results will also be summarized in this paper.


## 4. RELM Test

In order to test methods for forecasting future earthquakes the Southern California Earthquake Center (SCEC) formed the working group for Regional Earthquake Likelihood Models (RELM) in 2000 [12]. For the first time a competitive test of prospective earthquake forecasts was to be carried out. Research groups were encouraged to submit forecasts of future earthquakes in California. At the end of the test period, the forecasts would be compared with the actual earthquakes that occurred.The ground rules for the RELM test were as follows.(1) The test region to be studied was the state of California; however the selected region extended somewhat beyond the boundaries of the state as shown in Figure1.(2) The objective was to forecast the largest earthquakes for which a reasonable number could be expected to occur in a reasonable time period. A five-year time period for the test was selected extending from 1 January 2006 to 31 December 2010. Earthquakes withM≥5 were to be forecast. This magnitude cutoff was chosen because at least 20 M≥5 earthquakes could be expected in this period. For M≥6, only about 2 would be expected so the 5-year period would be much too short. The applicable magnitudes were taken from the Advanced National Seismic System (ANSS) online catalog (http://www.ncedc.org/anss/anss-detail.html).(3) Participants were required to submit the number of earthquakes expected to occur in specified spatial cells and magnitude bins during the test period. In order to do this, the test region was subdivided intoNc=7682 spatial cells with dimensions 0.1°×0.1° (approximately 10 km × 10 km). These spatial cells were further divided into 41 magnitude bins: 4.95≤M<5.05, 5.05≤M<5.15, 5.15≤M<5.25,…, 8.85≤M<8.95, and 8.95≤M<∞. The participants were required to specify the forecast number of earthquakes Nfmi in magnitude bin m (m-0.05<M<m+0.05) that would occur during the test period in cell i.It is important to note that the RELM forecasts were continuous (probabilistic) rather than alarm based. The numbers of earthquakes expected to occur in each spatial cell and each magnitude bin was required. Continuous and alarm-based forecasts each have advantages and disadvantages. Continuous forecasts are useful for setting insurance premiums but the numbers of predicted earthquakes are so small that they have little meaning to the general public. Alarm-based forecasts specify where earthquakes are most likely to occur.Nineteen forecasts were submitted by eight groups. Before discussing these forecasts in some detail we will discuss the earthquakes that occurred in the test region during the test period withM≥4.95.


## 5. The Earthquakes

During the test period 1 January 2006 to 31 December 2010, there wereNe=31 earthquakes in the test region with M≥4.95. The times of occurrence, locations, and magnitudes of these earthquakes are given in Table 1. The locations of the test earthquakes are also shown in Figure 1.Table 1
Times of occurrence, locations, and magnitudes of the 31 earthquakes in the test region withM≥4.95 from 1 January 2006 until 31 December 2010. The M=7.2 El Mayor-Cucapah earthquake is in bold.
No.Origin time (UTC)Lat.Long.M12006/05/24 04:20:26.0132.3067−115.22785.3722006/07/19 11:41:43.4640.2807−124.43325.0032007/02/26 12:19:54.4840.6428−124.86625.4042007/05/09 07:50:03.8340.3745−125.01625.2052007/06/25 02:32:24.6241.1155−124.82455.0062007/10/31 03:04:54.8137.4337−121.77435.4572008/02/09 07:12:04.5532.3595−115.27735.1082008/02/11 18:29:30.5332.3272−115.25685.1092008/02/12 04:32:39.2432.4475−115.31754.97102008/02/19 22:41:29.6632.4325−115.31305.01112008/04/26 06:40:10.6039.5253−119.92895.00122008/04/30 03:03:06.9040.8358−123.49685.40132008/07/29 18:42:15.7133.9530−117.76135.39142008/11/20 19:23:00.1932.3288−115.33184.98152008/12/06 04:18:42.8534.8133−116.41885.06162009/09/19 22:55:17.8432.3707−115.26125.08172009/10/01 10:01:24.6736.3878−117.85875.00182009/10/03 01:16:00.3136.3910−117.86085.19192009/12/30 18:48:57.3332.4640−115.18925.80202010/01/10 00:27:39.3240.6520−124.69256.50212010/02/04 20:20:21.9740.4123−124.96135.88222010/04/04 22:40:42.1532.2587−115.28727.20232010/04/04 22:50:17.0832.0972−115.04675.51242010/04/04 23:15:14.2432.3000−115.25955.43252010/04/04 23:25:06.9532.2462−115.29785.38262010/04/05 00:07:09.0732.0180−115.01725.32272010/04/05 03:15:24.4632.6282−115.80624.97282010/04/08 16:44:25.9232.2198−115.27605.29292010/06/15 04:26:58.4832.7002−115.92135.72302010/07/07 23:53:33.5333.4205−116.48875.43312010/09/14 10:52:18.0032.0485−115.19824.96The 31 earthquakes occurred inNce=22 cells. The association of earthquakes with cells is given in Table 2. Five of the 22 cells had multiple earthquakes. The occurrence of five test earthquakes in cell A is not surprising since this is in the Cerro Prieto geothermal area that is recognized as having a high level of seismicity. Earthquakes occurred in 22 of the 7682 0.1°×0.1° test cells in the test area.Table 2
Cell scoresλfi of an earthquake with M≥4.95 for the 22 cells in which earthquakes occurred during the test period. The association of cell IDs (A–V) with the earthquake IDs (1–31) from Table 1 is given. Five submitted forecasts are given: (1) Bird and Liu (B and L), (2) Ebel et al. (Ebel), (3) Helmstetter et al. (Helm.), (4) Holliday et al. (Holl.), and (5) Wiemer and Schorlemmer (W and S). The highest (best) scores are in bold.
Cell IDEQ IDB and LEbelHelm.Holl.W and S(A)1,7,8,16,241.99e-22.20e-21.17e-13.32e-21.24e-1(B)21.41e-23.40e-27.20e-23.32e-24.99e-2(C)37.40e-36.59e-37.41e-33.32e-27.91e-3(D)43.54e-23.29e-26.97e-23.32e-23.59e-2(E)57.23e-31.10e-32.29e-39.72e-51.58e-7(F)69.37e-32.85e-23.07e-23.32e-24.55e-2(G)9,109.11e-35.49e-32.55e-23.32e-22.38e-2(H)113.42e-45.49e-39.15e-41.62e-42.06e-4(I)122.14e-31.10e-33.65e-32.05e-49.89e-3(J)131.68e-38.78e-31.11e-23.32e-21.13e-2(K)143.12e-22.20e-23.30e-23.32e-25.90e-2(L)152.07e-35.49e-36.93e-33.32e-32.64e-3(M)17,181.74e-32.20e-35.78e-33.32e-25.38e-4(N)195.83e-26.59e-31.49e-23.32e-27.44e-3(O)201.25e-21.43e-29.45e-33.32e-21.62e-2(P)216.48e-33.29e-22.71e-23.32e-27.46e-3(Q)22,25,282.88e-22.20e-22.84e-23.32e-25.23e-2(R)23,263.06e-21.54e-21.43e-21.73e-41.58e-2(S)272.13e-25.49e-31.26e-23.32e-21.19e-2(T)291.83e-21.32e-22.43e-23.32e-24.99e-2(U)301.26e-23.07e-21.03e-13.32e-35.16e-2(V)316.76e-31.54e-25.55e-33.32e-22.64e-3The major earthquake that occurred during the test period was theM=7.2 El Mayor-Cucapah earthquake on 4 April 2010 (event 22 in Table 1). This earthquake was on the plate boundary between the North American and Pacific plates. The epicenter was about 50 km south of the Mexico-United States border, but occurred within the test region as shown in Figure 1. Events 23, 24, 25, 26, 27, 28, 29, and 31 are well-defined aftershocks of the El Mayor-Cucapah earthquake. Events 1, 7, 8, 9, 10, 14, 16, and 19 constitute a precursory swarm of eight test earthquakes in this region in the magnitude range 4.97 to 5.80, including four in the 10-day period between 9 February and 19 February 2008 (events 7–10). These events were located some 5 km to 20 km north of the subsequent epicenter of the El Mayor-Cucapah earthquake and lie outside the primary aftershock region of that event. This swarm of earthquakes certainly cannot be considered foreshocks due to their relatively small magnitudes and early occurrence but may represent a seismic activation. We will discuss this activation in terms of AMR later in this paper.Another swarm of earthquakes occurred in the northwest corner of the test region adjacent to Cape Mendocino. This sequence (events 23, 4, 5, 20, and 21) had magnitudes in the range 5.0 to 6.5. This is a region of high seismicity, and this concentration of events is expected. Event 21 may or may not be an aftershock of event 20. The pair of earthquakes 17 and 18 are interesting. It is very likely that theM=5.0 earthquake on 1 October 2009 was a foreshock of the M=5.19 earthquake on 3 October 2009.


## 6. Submitted Forecasts

The submitted forecasts have been discussed in some detail [13]. The nineteen forecasts submitted by eight groups are available on the RELM website (http://relm.cseptesting.org/). In order to have a common basis for comparison, we will only consider forecasts that cover the entire test region. Seven forecasts were submitted that gave the predicted number, Nfmi, for M≥4.95 earthquakes in 0.1 magnitude bins during the five-year test period for all Nc=76820.1°×0.1° cells.The submitted forecasts are based on a variety of approaches. The Bird and Liu forecast [14] was based on a kinematic model of neotectonics. The Ebel et al. forecast [15] was based on the average rate of M≥5 earthquakes in 3°×3° cells for the period 1932 to 2004. The Helmstetter et al. forecast [16] was based on the extrapolation of past seismicity. The Holliday et al. forecast [17] was based on the extrapolation of past seismicity using a modification of the pattern informatics (PI) technique. The Wiemer and Schorlemmer forecast [18] was based on the asperity-based likelihood model (ALM).We will now discuss the Holliday et al. forecast in somewhat greater detail. The basis of this RELM forecast followed the format introduced in the PI forecast methodology [7, 8]. The magnitude range M≥5 and the cell dimensions 0.1°×0.1° were the same. However, the PI method was alarm based. Earthquakes were forecast to either occur or not occur in specified regions (hotspots) in a specified time period. In the PI-based RELM forecast, all hotspot cells are given equal probabilities of an earthquake. For the values in Table 2, λfi=3.32×10-2. Instead of being alarm based, the RELM test was based on probabilities of occurrence of an earthquake in each cell in the test region. This required a continuous assessment of risk rather than a binary, alarm-based assessment. To do this, the Holliday et al. [17] forecast introduced a uniform probability of occurrence for hotspot regions and added smaller probabilities for nonhotspot regions based on the relative intensity (RI) of seismicity in the region. A map of the Holliday et al. [17] forecast is given in Figure 2.Figure 2
Map of the normalized probabilitiesλfi given for the testregion by Holliday et al. [17] using their PI-based forecast. The “hotspots” are shown in red. The test earthquakes are also shown.As stated in our description of the RELM test, each participant submitted a forecast for the number of earthquakesNfmi in magnitude bin m that would occur in cell i. Thus 41×7682=314962 values of Nfmi were submitted in each forecast. In order to better understand the implications of the forecasts we sum the probabilities in the magnitude bins for each spatial cell to give the number of forecast earthquakes Nfi in cell i with magnitude M≥4.95:(2)Nfi=∑m=59Nfmi.
The reason we carry out this sum is so that we can directly apply the “skill score” methodology developed in the atmospheric sciences. In terms of forecasting tornadoes, the question is whether a tornado occurs, not its strength. Since the RELM test was for earthquakes with M≥4.95 our scoring is whether such an earthquake occurs or does not occur in a spacial cell.The sum of theNfi over all cells is the total number of earthquakes Nf with M≥4.95 forecast to occur during the test period:(3)Nf=∑i=1NcNfi,
where Nc is the total number of cells. Our objective is to separate the forecast of the total number of earthquakes from the forecast of their locations. In order to do this we introduce a cell score λfi defined by(4)λfi=NceNfiNf,
where Nce is the number of cells in which an earthquake occurred during the test period. Note that from (3) and (4) we have(5)∑i=1Ncλfi=Nce.
Thus, the sum of λfi over all cells is the same for each submitted forecast. The cell score λfi is a direct measure of the probability of occurrence of a test earthquake in cell i. A perfect forecast (a perfect skill score) would have λfi≥1 for the cells in which earthquakes occur and λfi=0 for all other cells. In principal λfi can be as big as Nce. However, because we are only concerned with whether an earthquake occurs in a cell, not how many occur—a point we discuss in the next paragraph—all values of λfi>1 are just treated as 1 for that particular cell. In practice this does not occur due to the small values of Nfmi provided by the RELM forecasts.Since the forecasts are for specific0.1°×0.1° cells, it is necessary to consider how to handle the forecasts when more than one earthquake occurs in a cell. As stated above, in our analysis a cell in which more than one earthquake occurred is treated the same as a cell in which only one earthquake occurred. This follows the practice used in tornado forecasting. How many tornadoes occur in a region during the forecast period is not considered, only whether one or more occur. For the test earthquakes given in Table 1, events 1, 7, 8, 16, and 24 occurred in the same cell, similarly for events 9 and 10, events 17 and 18, events 22, 25, and 28, and events 23 and 26. This multiplicity is shown in Table 2. Thus, we will consider forecasts made for 22 cells.Taking the actual number of cells in which earthquakes occurred to beNce=22 and the total number of earthquakes forecast in each submission Nf using (3), we obtained the forecast scores λfi using (4).The seven submitted forecasts included two submissions with separate forecasts with and without aftershocks. Different numbers of events were forecast but the relative scores of locations were the same. Thus, we consider five submissions. The forecast scoresλfi for each of the five submissions are given in Table 2 for the Nce=22 cells in which an earthquake occurred. A perfect forecast in which only the 22 cells were forecast to have earthquakes would have λfi=1 in each of the 22 cells. A random forecast in which all Nc=7682 cells were given the same Nfi=a would yield(6)λfirandom=NceaNf=NceNc=227682=2.86×10-3.
The submitted forecast scores in Table 2 have a wide range of values from λfi=1.58×10-7 to λfi=1.24×10-1.


## 7. Evaluation of Results

During the formulation of the RELM project a comprehensive testing strategy was also developed [19]. A suite of likelihood tests were proposed, which would be implemented through a testing center [20]. The approach utilized an L-test, an N-test, and an R-test. These tests were applied to the raw submitted data. This approach was applied to the first 2.5 years of RELM results by Schorlemmer et al. [13]. Zechar et al. [21] recognized a problem with the original proposed likelihood tests and proposed a modification.This is certainly one approach to the evaluation of results, the primary purpose of this paper is to present a complementary approach. Our approach has the advantage that the evaluation of the numbers of earthquakes forecast can be separated from the forecast of their locations.Lee et al. [22] proposed the modified approach to the evaluation of RELM test results that is used in this paper. In their short paper they compared the forecasts that had been submitted for all of California. In this paper we consider a subset of those forecasts and relate the results to the concept of alarm-based forecasts.The results given in Table2 can be used to compare the forecast scores for each of the cells in which earthquakes occurred. The highest scores between the models are shown in bold. Clearly there are many ways in which to evaluate the results of the forecasts. There is a tradeoff between good forecasts with large λfi and poor forecasts with small λfi. We first consider the forecasts that had the highest forecast scores. The Holliday et al. [17] forecast had the largest λfi for 8 of the 22 cells in which (target) earthquakes occurred. The Wiemer and Schorlemmer [18] forecast had 6 of the largest λfi. The Helmstetter et al. [16] forecast had 4 of the largest λfi. Finally, the Bird and Liu [14] forecast had 3 of the largest λfi. These values are also given in Table 3. The range of the highest cell scores was from λfi=1.24×10-1 for event 1 to λfi=5.49×10-3 for event 11.Table 3
Comparisons of the forecasts: Column 1. the number of maximum cell scoresNλmax. Column 2: the mean cell scores forecast λ̅f. Column 3: the number of earthquakes Nf predicted by each forecast. The best scores in each category are in bold.
Nλmaxλ̅fNfBird and Liu31.53e-256Ebel et al.11.51e-2115Helmstetter et al.42.84e-235Holliday et al.82.45e-230Wiemer and Schor.62.66e-224It is also of interest to compare the mean cell forecast scores for the 22 cells in which earthquakes occurred. These valuesλ̅f are given in Table 3. The Helmstetter et al. [16] forecast had the highest λ̅f=2.84×10-2, the Wiemer and Schorlemmer [18] forecast had λ̅f=2.66×10-2, and the Holliday et al. [17] forecast had λ̅f=2.45×10-2. The Helmstetter et al. [16] forecast did the best in an average sense but did relatively poorly in providing the best cell forecasts. It should be noted that the best average forecast λ̅f=2.84×10-2 is one order of magnitude better than the random (no skill) forecast λfirandom=2.86×10-3.As noted above, the Holliday et al. [17] forecast is primarily an alarm-based (hotspot) forecast. The PI method was used to determine the cells in which earthquakes were most likely to occur (hotspots). In the cell forecasts given in Table 2, these cells had forecast scores λfi=3.32×10-2 and consisted of 8.3% of the total area of the test region (637 of the 7682 cells). Of the 22 cells in which earthquakes occurred, 17 occurred in hotspot cells. In 8 of the 17 cells, the forecast cell scores given by the Holliday et al. [17] forecast were the highest.


## 8. Discussion

The RELM test provides a well-defined set of prospective earthquake forecasts and a well-defined set of test earthquakes. In this paper we present a method for evaluating the RELM forecasts. We believe our approach has significant advantages but look forward to comparing our results with those obtained by other authors.RELM forecasts provide the numbersNfmi of earthquakes expected to occur in magnitude bins m and spatial cells i. The basis of our approach is(1)
to use (2) to determine the forecast number Nfi of earthquakes with M≥4.95 expected to occur in spatial cell i,(2)
to use (3) to determine the total forecast number Nf of earthquakes,(3)
to use (4) to determine the cell score λfi.We first compared the actual number of earthquakes that occurred during the test period, 31 with the forecast values. The closest forecast values were those of Holliday et al. [17] with Nf=30 as shown in Table 3.We next compared the forecast scoresλfi of an earthquake with M≥4.95 occurring in cell i. We noted that the values of λfi were the same for the two submissions in which both main shocks and aftershocks plus main shocks were submitted. These forecasts gave different values for the numbers Nfmi, Nfi, and Nf of earthquakes but the forecast distributions in space were identical.In a perfect forecast the forecast score would have beenλfi=1 for each of the 22 cells in which one or more earthquakes occurred and λfi=0 in the other 7660 cells. The mean forecast scores for the 22 cells in which earthquakes occurred for the five forecasts ranged from a high value λ̅f=2.84×10-2 to a low value of λ̅f=1.53×10-2. The range of values was relatively small, about a factor of two. The random (no skill) forecast assuming equal probabilities for the 7682 cells in the test region gives a forecast score λfirandom=2.86×10-3 for all cells. The best forecast score λfi=2.84×10-2 was about a factor of 10 better than the random forecast but a factor of 100 worse than a perfect forecast.As we have previously discussed earthquake forecasts can be either probabilistic or alarm based. The submission rules for RELM were probabilistic. The only forecast that had an alarm-based distribution of forecasts was that of Holliday et al. [17]. A question of interest for future tests of earthquake forecasts is whether they should be alarm or probability based. A systematic study of alarm-based forecasts could be of considerable interest.Another interesting question is whether the forecasts have a temporal component. Is there a time-dependent component in the data used that changes forecast probabilities significantly? As discussed previously, eight of the test earthquakes were aftershocks of the El Mayor-Cucapah earthquake and eight of the test earthquakes were associated with a precursory swarm. Thus, 17 of the 31 of the test earthquakes were associated with this earthquake. It appears reasonable to conclude that precursory activation prior to the El Mayor-Cucapah earthquake may have played a significant role in the success of forecasts.Another swarm of 6 earthquakes during the test period adjacent to Cape Mendocino did not lead to a subsequent larger event during the test period. Swarms of activity in this region occur regularly. In terms of precursory activation this activity would lead to a false alarm. The contrast between the two regions (Cape Mendocino and El Mayor-Cucapah) is an indication of the difficulties in forecasting earthquakes utilizing precursory activation.



---

*Source: 543482-2012-03-20.xml*