Analysis on selected geo-effective
events using observations and models at
Space Environment Prediction Center
Siqing Liu, Ercha Aa, Qiuzhen Zhong, Bingxian Luo,Zhitao Li, Jingjing Wang, and Jiancun Gong
Space Environment Prediction Center,
National Space Science Center, Chinese Academy of Sciences

SEPC (http://eng.sepc.ac.cn)
 Established in 1992 to support space exploration
 Started to issue space environment prediction in 1998
 7 days/week & 365 days/year
Chinese Academy of Sciences

National Space Science CenterSEPC

Space Weather Forecasting Services
 SEPC is now an Associate Warning Center of The International
Space Environment Service (ISES).
General ProductsSpace Weather Indices

Ap, Kp, F107, Dst, AE/AU/AL…

Space Weather Events

Solar Flares, SPE, Geomagnetic Storm,
Relativistic Electron Enhance…

Tailored Products

Manned Space Mission, Lunar Exploration…

Publishing Ways

Website, Text Message, Mail, Email, Mobile App…

Outline Overview of the Space Environment
 24th Solar Cycle Year 2014

 Verification of SEPC/NSSC Forecasting Single Storm Event Jun, 2014 - Jan, 2015

Smoothed sunspot #Overview of the 24th Solar Cycle
Sunspot # from 1st-24th solar cycle

Two Peak profile has shown

Entering Decreasing Phase

Estimated Total Sunspot #: ~4400

74%
1st Peak

2nd PeakFeb, 2012

Apr, 2014

2014 Overview: Solar Activity
2013

2014

No sunspot day

0

1

Active Region #302

319（+6%）

Average Sunspot # 97

121（+25%）

Mean F10.7

123

146（+19%）

C-class Flares

1357

1798（+32%）

M-class Flares100

208（+108%）

X-class Flares

1216（+33%）

SEP Events

76（-14%）2014 Oct-Nov (AR2192: 35M- & 6 X-class flares)

Active Region with
Maximum Area inthe 24th solar cycle

X1.1

X1.6

X3.1
X1.0

X2.0

X2.0

2014 Overview: Geomagnetic Activity

Average AP# of Days for AP≥15
Geomagnetic Active Days
Minor/Moderate Storm Days
Strong Storm Days
Maximum AP
Maximum Kp

201384115449
151
7

KpValues

Geomagnetic
Activity level

Kp = 4ActiveKp = 5

Minor Storm

Kp = 6Moderate Storm

Kp = 7/8

Strong Storm

Kp = 9Severe Storm2014
9（+13%）
49（+20%）170（+10%）47（-4%）1（0%）
47（-8%）
7（0%）

Outline Overview of the Space Environment
 24th Solar Cycle Year 2014

 Verification of SEPC/NSSC Forecasting Single Storm Event Jun, 2014 - Jan, 2015

SEPC Space Weather Forecasting Products (Index/Flux)Geomagnetic Storm Lists from Jun, 2014-Jan, 2015
Time (Y/M/D)

Storm Level/Duration

Causes Description

2014/06/08-06/09

Minor/ 3 hours

CME from Filament

2014/06/18

Minor/ 3 hours

CIR

2014/08/19

Moderate/ 3 hours

CME from Filament

2014/08/27-08/31

Minor/ 6 hours

CME + CIR

2014/09/13-09/14

Strong/ 3h+ Moderate/ 3h + Minor/ 9h

CME from AR2158, X1.6 flare

2014/09/19

Minor/ 3 hours

CME from AR2157 + following CIR

2014/10/14-10/15

Minor/ 9 hours

CME from filament

2014/10/20-10/22

Minor/ 3 hours

CIR

2014/11/10-11/12

Minor/ 6 hours

CME from AR2205 + following CIR

2014/12/06-12/09

Minor/ 6 hours

CIR

2014/12/12

Minor/ 3 hours

CIR

2014/12/21-12/24

Minor/ 6 hours

CME from AR2242 + following CIR

2014/12/28-12/30

Minor/ 6 hours

CIR

2015/01/02-01/03

Minor/ 3 hours

CME from filament

2015/01/04-01/05

Minor/ 9 hours

CME

2015/01/07-01/08

Strong/3h + Moderate/3h

CME

Remarks

Storm Levels
Minor: Kp=5
Moderate: Kp=6
Strong: Kp=7/8
Severe: Kp = 9

Geomagnetic storms @ Aug 27-31, 20146-h Storm caused by CMEs from Aug 22 with
C6 flare from AR2148 or C2/C6 flares from AR2146
Following with multiple CIRs

Forecasting Verification Factors
FactorFormulation

Reflection

Mean Error (ME)

1 N(Fi − Oi )=ME∑
N i =1Bias

Mean Absolute Error (MAE)

1 N|Fi − Oi |=MAE∑
N i =1Accuracy

1 N2(F)−O∑ i iN i =1Accuracy

Root Mean Square Error (RMSE)

=RMSE
N

Correlation Coefficient (CC)CC =∑ (F − F )(O − O)i =1iNiN∑ (Fi − F )2∑ (Oi − O)2

MSESS = 1 −MSE forecast

Association=i 1 =i 1MSE Skill Score (MSESS or PE)

MSEreference

SkillGeomagnetic Storm@ Aug. 27-31, 2014 : Ap forecasting results
Model Description
 Ap forecasting for future 27-days using auto-regression analysis model
Ap modAp art

Reflection

ME-6.583-2.75

BiasMAE

6.75

4.083

RMSE

8.39

5.346

CC

0.716

0.634

Association

MSESS

-1.00

0.186

Skill

Better OneModArt√√

Accuracy√√√

Evaluation

Ap model forecasting results

Ap artificial forecasting results

What is theforecasting
performance

Larger bias during storm time
Fair accuracy agreement
Higher correlation coefficients but lower skills

Smaller bias during storm time
Better accuracy agreement
Lower correlation coefficients & Higher skills

Why such
Performance

Focus on the general consistence of the 27days prediction with details being neglected

More flexible reaction & adjustment of people basedon the level and duration of the current storm

Where it canbe used andhow to use

Where: Space Weather prediction, research and application, and SEPC/NSSC online forecasting tools
How: Forecasting results (future 27-days by model and 3 days by artificial) are updated and publishedon the website & mobile App simultaneously, and can be browsed & downloaded conveniently by users.

Geomagnetic Storm@ Aug. 27-31, 2014 : F10.7 forecasting results
Model Description
 F10.7 forecasting for future 27-days using auto-regression analysis model with 54-orders
F10.7modF10.7
art

Reflection

ME-0.583

0.75

Bias

MAE

7.583

6.083

RMSE

10.571

9.543

CC

0.781

0.748

AssociationMSESS

0.011

0.194

Skill

Better OneModArt√√

Accuracy√√√

Evaluation

F10.7 model forecasting results

F10.7 artificial forecasting results

What is theforecasting
performance

Comparable bias between mod and art results
Fair accuracy agreement (~7-8% relative errors)
Higher correlation coefficients but lower skills
1-day delay in grasping inflection point

Why such
Performance

Generally stable variation trends, which is readily to grab
Historical periodical data cannot reflect the sudden appearance/disappearance of active region

Where it canbe used andhow to use

Where: Space weather prediction, and empirical/theoretical models with F10.7 as driven factor
How: Forecasting results (future 27-days by model and 3 days by artificial) are updated and publishedon the website & mobile App simultaneously, and can be browsed & downloaded conveniently by users.

Comparable bias between mod and art results
Fair accuracy agreement (~7%-8% relative errors)
Lower correlation coefficients & moderate skills
1-day delay in grasping inflection point

Geomagnetic Storm@ Aug. 27-31, 2014 : Dst forecasting results
Model Description
 1-h forecasting of Dst index using real time solar wind data from ACE observation
based on the influence of Sw dynamic pressure to the decay & injection of ring current
Dst modReflection

ME-9.321

BiasMAE

10.243

RMSE

11.665

CC0.946

AssociationVery HighMSESS0.678

SkillHigh

Evaluation

Accuracy

Evaluation

10%-15%

Dst model forecasting results

What is theforecastingperformance

Good accuracy agreement (~10-15% relative errors)
High correlation coefficients and skills (close to 1)

Why such
Performance

The modeled Dst is a sum of three terms that have growth and decay, a dynamic pressure term, aninterplanetary magnetic field term, and some offset terms.

Where it canbe used andhow to use

Where: Space weather prediction, , operational system in SEPC/NSSC, and web-based forecasting tools
How: Forecasting results (future 1 hours) are updated and published on the website & mobile App
simultaneously, and can be browsed & downloaded conveniently by users.

Geomagnetic Storm@ Aug. 27-31, 2014 : Kp forecasting results
Model Description
 1-hour in advance forecasting of Kp index using real time solar wind data from ACE
observation based on neural network algorithms.
Kp modReflection

ME-0.83

Bias

MAE

0.897

RMSE

1.13

CC

0.813

AssociationHigh

MSESS

0.340

Skill

Moderate

Evaluation

Accuracy

Evaluation~20%-30%

Kp model forecasting results

What is theforecasting
performance

Commonly Underestimation of Kp index around 20%-30%
High correlation coefficients
Moderate skill score

Why such
Performance

Relative stable and subtle fluctuation of the geomagnetic field for the 24th solar cycle, which as beenused as the background values of the input of neural network

Where it canbe used andhow to use

Where: Space weather prediction, geomagnetic storm alert and warning
How: Forecasting results (1-hour in advance) are updated and published on the website & mobile App
simultaneously, and can be browsed & downloaded conveniently by users.

Geomagnetic Storm@ Aug. 27-31, 2014 : Relativistic Electron Forecasting
Model Description
 1-day in advance forecasting of daily integrated relativistic electron flux > 2Mev at
geosynchronous orbit using linear prediction combined with Kalman filter
ModelReflection

Evaluation

ME-0.157

Bias3%

MAE

0.462

RMSE

0.682

Accuracy~8%-10%CC

0.861

AssociationHigh

MSESS

0.436

SkillHigh

Evaluation

Electron flux model forecasting results

What is theforecasting
performance

Sometimes lagged by 24 hours in grasping the sudden variation of the values
Less than 10% prediction errors
High correlation coefficients and skills

Why such
Performance

Restriction of linear prediction in representing sudden variation

Where it canbe used andhow to use

Where: Space weather prediction, aircraft alert and warning at GEO orbit
How: Forecasting results (1-day in advance) are updated and published on the website & mobile App
simultaneously, and can be browsed & downloaded conveniently by users.

Geomagnetic Storm@ Aug. 27-31, 2014 : AE Index Forecasting
Model Description
 1-hour in advance forecasting of AE Index with 10 min resolution based on empiricalfunctions by using solar wind data, IMF parameters, and F10.7 index as input
ModelReflection

Evaluation

ME2.43

Bias1%-2%

MAE

43.65

RMSE

75.83

Accuracy~15%-20%CC

0.84

AssociationHighMSESS

0.81

SkillHigh

Evaluation

Electron flux model forecasting results

What is theforecasting
performance

Very good performance in grabbing the trends of the variation and the level of the peak values
15%-20% prediction errors
High correlation coefficients and skills

Why such
Performance

Proper settings of the empirical function in establishing the relationship between solar wind andelectro-jet in polar region

Where it canbe used andhow to use

Where: Space weather prediction, or driven factor for other empirical/theoretical models
How: Forecasting results (1-hour in advance and 10 min resolution) are updated and published on thewebsite & mobile App simultaneously, and can be browsed & downloaded conveniently by users.

Verification of SEPC ForecastingJun 2014 - Jan 2015

Verification of SEPC Forecasting: Accuracy
 How to choose appropriate factor to represent forecasting accuracy?
 RMSE for index with low base values, and NRMSE for high base values=RMSE

1 N(Fi − Oi ) 2∑N i =1

NRMSE =

1 N Fi − Oi 2∑( O )N i =1i

Ap Model6.76--

Ap Artificial

5.87--

Kp

1.19--

Dst

14.37

--

F10.7 Model

7.11

4.89%

F10.7 Artificial

7.60

5.22%

AE

72.03

32.43%

0.47 (after log10)

6.54%

Relativistic Electron

Verification of SEPC Forecasting: Association and Skill
Correlation Coefficients
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
00.968 0.968

0.891

0.871

0.717
0.593

0.524

Relative high correlation coefficientsfor various parameters except for Apindex model0.283ApmodAp art F107
modF107art

Dste_flux

Kp

Evaluation: AssociationAE

Skill Score (PE)
1.000
0.800
0.600
0.400
0.200
0.000
-0.200-0.400Evaluation: Skill Score
Relative high skill score for F10.7,
Relativistic electron, and AE index
comparing with persistence model.
Ap Ap art F107 F107
Dst e_flux Kp
AEmodmodart-0.278 0.037 0.940 0.932 0.120 0.803 -0.325 0.780

Summary and Conclusions
 Overview of the 24th solar cycle
 The current 24th solar cycle has a relative small activity level, and the solar activity has reachedthe highest point and began to decrease.
 Overview of 2014
 The solar activity is stronger than that of the year before. The geomagnetic activity remains
almost the same. The average Ap levels and geomagnetic active days are increased, but thestorm days are almost the same with the year before.
 Verification of SEPC forecasting results
 The forecasting results of Ap, Kp, F10.7 , Dst, Relativistic Electron, and AE index have beenverified for selected event and for period of Jun 2013-Jan 2014 based on the bias, accuracy,
association, and skill respectively. The performance of the model as well as the artificial
forecast have been evaluated accordingly. The forecasting results can be used for space weather
prediction, operational system in SEPC/NSSC, and web-based forecasting tools. The
forecasting results are updated and published on the website & mobile App simultaneously, andcan be browsed & downloaded conveniently by users.

Thanks for your attention!

Space Environment Prediction Center, NSSC/CAS