tograms. The ARs were selected between longitudes of 70° E/W on the solar disk and the effective magnetic field ($B_{eff}$) was calculated for each. Due to limb effects, a proxy was used to find $B_{eff}$ between 50° to 70° E/W. The probability of flare occurrence from C, M, and X-class flares was derived for each $B_{eff}$. The likelihood of a flare of a particular peak SXR flux to erupt with a CME was inferred from Yashiro et al. (2005). Folding the two probability functions gives the likelihood an AR will produce a CME. Additionally, the CME speed is predicted using a relationship with $B_{eff}$ derived in Georgoulis (2008): $V_{CME} = 87.3B_{eff}^{0.38}$ [km/s].

A separate SEP forecasting module was derived from a database of SEPs (>10, >30, >60, and >100 MeV), flares, CMEs and radio fluxes compiled by Papaioannou et al. (2016) spanning three decades from 1984 to 2013. The solar disk was divided into five longitudinal bins and 4000 flares in each bin were used to derive the probability of SEP occurrence per SF magnitude at each longitude: $P(SEP_{flare})_i = (N_{SEP})_i/(N_{all})_i$. For each AR location, the probability that an SEP will occur per SF magnitude is multiplied by the probability distribution output by the flare forecasting module that a flare of a given magnitude will be produced by the AR. This produces the final output from the forecasting module: the maximum probability of SEP occurrence for each AR (Anastasiadis et al., 2017).

After a SF has occurred, the FORSPEF flare nowcasting module uses information from Solar DEMON to make SEP predictions. Probability distribution functions for the observed flare longitude are calculated from the FORSPEF database and the SEP probability of occurrence is identified following a similar approach to the forecasting mode. The module also reports the expected SEP values derived from the SXR peak flux, e.g., proton peak flux, time of peak, duration, and fluence for each energy channel (Papaioannou et al., 2016).

FORSPEF also forecasts SEP characteristics from CMEs observed by the automated real time CACTUS CME catalog. CACTUS is the only near real-time solution that provides reliable and continuous outputs that can be used for SEP nowcasting (Anastasiadis et al., 2017; Papaioannou et al., 2018). To develop the CME to SEP nowcasting tool, however, SEP event information from the FORSPEF database (Papaioannou et al., 2016) was paired with CME information from the CDAW CME Catalog 8. Data from 1997–2013 were used, representing the overlapping time period between the two catalogs, comprising a total of 1905 CMEs and 158 SEP events (>10 MeV).

The CMEs were sorted into a matrix of nine subgroups according to their speed and width. The speed ranges were divided into slow (400 km/s ≤ V < 1000 km/s), moderate (1000 km/s ≤ V < 1500 km/s) and fast (V ≤ 1500 km/s) and the Angular Width (AW) ranges divided into full halo

(AW = 360°), partial halo (120° ≤ AW ≤ 359°) and non halo (AW < 120°). The probability that an SEP event is produced by a CME in each subgroup was calculated along with the correspond 68% confidence level. For each CME angular width category, curves were fitted to the SEP probabilities to derive continuous functions of SEP probability with velocity for halo, partial halo, and non halo CMEs (Papaioannou et al., 2018).

To calculate the relationship between the CME parameters and SEP peak flux and total fluence, linear regressions were performed between the CME speed and peak flux or fluence in each energy channel ($E > 10, > 30, > 60, and > 100$ MeV). Separate linear regressions were found for halo, partial halo, and non-halo CMEs with each GOES energy channel. Tables specifying the linear regressions and their standard error are specified in (Papaioannou et al., 2018).

A third nowcasting module incorporates the ESPERTA model (Section 3.8) approach to predict whether an SEP event will exceed S1 on the NOAA Solar Storm Scale. Radio data at 0.944 MHz from the Wind/WAVES instrument were compiled and calculations of the radio and SXR data were refined. Revised versions of the SEP occurrence equations in ESPERTA were produced. However, the Wind/WAVES data or an appropriate substitute is not available in near-real time, so this module cannot presently be used in the operational version of FORSPEF.

**Inputs:**

**Forecasting:** 1) SDO/HMI full-disk magnetograms, 2) AR number from NOAA/SWPC, 3) FORSPEF database for SEPs, SFs, CMEs (Papaioannou et al., 2016).

**Nowcasting:** 1) near real-time SF data from Solar DEMON 9, 2) near real-time CME identifications obtained from the automated CACTUS CME catalog 10, 3) integrated radio flux at 1 MHz for the ESPERTA Model (not available in real-time).

Both forecasting and nowcasting modes use the FORSPEF database for SEPs, SFs, CMEs (Papaioannou et al., 2016).

**Outputs:** Forecasts on the FORSPEF web page provide the AR NOAA number, AR heliographic and device-coordinate location, the corresponding $B_{eff}$ value, the 24-h cumulative flare probabilities for each of the 28 GOES flare classes (from C1.0 to X10.0) and their peak photon fluxes, the respective eruptive-flare or CME likelihoods, the projected CME velocity, and the probability of SEP occurrence.

Nowcasts report the probability of SEP occurrence and SEP characteristics - onset time, peak time, decay time, peak flux, and fluence for ($E > 10, > 30, > 60, and > 100$ MeV) in table and graphical outputs.

**Free parameters:** None.

<footnote n="8">8 https://cdaw.gsfc.nasa.gov/CME_list/.</footnote>

<footnote n="9">9 http://solardemon.oma.be/flares.php.</footnote>

<footnote n="10">10 https://wwwbis sidc.be/cactus/.</footnote>
tograms. The ARs were selected between longitudes of 70° E/W on the solar disk and the effective magnetic field ($B_{eff}$) was calculated for each. Due to limb effects, a proxy was used to find $B_{eff}$ between 50° to 70° E/W. The probability of flare occurrence from C, M, and X-class flares was derived for each $B_{eff}$. The likelihood of a flare of a particular peak SXR flux to erupt with a CME was inferred from Yashiro et al. (2005). Folding the two probability functions gives the likelihood an AR will produce a CME. Additionally, the CME speed is predicted using a relationship with $B_{eff}$ derived in Georgoulis (2008): $V_{CME} = 87.3B_{eff}^{0.38}$ [km/s].

A separate SEP forecasting module was derived from a database of SEPs (>10, >30, >60, and >100 MeV), flares, CMEs and radio fluxes compiled by Papaioannou et al. (2016) spanning three decades from 1984 to 2013. The solar disk was divided into five longitudinal bins and 4000 flares in each bin were used to derive the probability of SEP occurrence per SF magnitude at each longitude: $P(SEP_{flare})_i = (N_{SEP})_i/(N_{all})_i$. For each AR location, the probability that an SEP will occur per SF magnitude is multiplied by the probability distribution output by the flare forecasting module that a flare of a given magnitude will be produced by the AR. This produces the final output from the forecasting module: the maximum probability of SEP occurrence for each AR (Anastasiadis et al., 2017).

After a SF has occurred, the FORSPEF flare nowcasting module uses information from Solar DEMON to make SEP predictions. Probability distribution functions for the observed flare longitude are calculated from the FORSPEF database and the SEP probability of occurrence is identified following a similar approach to the forecasting mode. The module also reports the expected SEP values derived from the SXR peak flux, e.g., proton peak flux, time of peak, duration, and fluence for each energy channel (Papaioannou et al., 2016).

FORSPEF also forecasts SEP characteristics from CMEs observed by the automated real time CACTUS CME catalog. CACTUS is the only near real-time solution that provides reliable and continuous outputs that can be used for SEP nowcasting (Anastasiadis et al., 2017; Papaioannou et al., 2018). To develop the CME to SEP nowcasting tool, however, SEP event information from the FORSPEF database (Papaioannou et al., 2016) was paired with CME information from the CDAW CME Catalog 8. Data from 1997–2013 were used, representing the overlapping time period between the two catalogs, comprising a total of 1905 CMEs and 158 SEP events (>10 MeV).

The CMEs were sorted into a matrix of nine subgroups according to their speed and width. The speed ranges were divided into slow (400 km/s ≤ V < 1000 km/s), moderate (1000 km/s ≤ V < 1500 km/s) and fast (V ≤ 1500 km/s) and the Angular Width (AW) ranges divided into full halo

(AW = 360°), partial halo (120° ≤ AW ≤ 359°) and non halo (AW < 120°). The probability that an SEP event is produced by a CME in each subgroup was calculated along with the correspond 68% confidence level. For each CME angular width category, curves were fitted to the SEP probabilities to derive continuous functions of SEP probability with velocity for halo, partial halo, and non halo CMEs (Papaioannou et al., 2018).

To calculate the relationship between the CME parameters and SEP peak flux and total fluence, linear regressions were performed between the CME speed and peak flux or fluence in each energy channel (E > 10, > 30, > 60, and > 100 MeV). Separate linear regressions were found for halo, partial halo, and non-halo CMEs with each GOES energy channel. Tables specifying the linear regressions and their standard error are specified in (Papaioannou et al., 2018).

A third nowcasting module incorporates the ESPERTA model (Section 3.8) approach to predict whether an SEP event will exceed S1 on the NOAA Solar Storm Scale. Radio data at 0.944 MHz from the Wind/WAVES instrument were compiled and calculations of the radio and SXR data were refined. Revised versions of the SEP occurrence equations in ESPERTA were produced. However, the Wind/WAVES data or an appropriate substitute is not available in near-real time, so this module cannot presently be used in the operational version of FORSPEF.

**Inputs:**

**Forecasting:** 1) SDO/HMI full-disk magnetograms, 2) AR number from NOAA/SWPC, 3) FORSPEF database for SEPs, SFs, CMEs (Papaioannou et al., 2016).

**Nowcasting:** 1) near real-time SF data from Solar DEMON 9, 2) near real-time CME identifications obtained from the automated CACTUS CME catalog 10, 3) integrated radio flux at 1 MHz for the ESPERTA Model (not available in real-time).

Both forecasting and nowcasting modes use the FORSPEF database for SEPs, SFs, CMEs (Papaioannou et al., 2016).

**Outputs:** Forecasts on the FORSPEF web page provide the AR NOAA number, AR heliographic and device-coordinate location, the corresponding $B_{eff}$ value, the 24-h cumulative flare probabilities for each of the 28 GOES flare classes (from C1.0 to X10.0) and their peak photon fluxes, the respective eruptive-flare or CME likelihoods, the projected CME velocity, and the probability of SEP occurrence.

Nowcasts report the probability of SEP occurrence and SEP characteristics - onset time, peak time, decay time, peak flux, and fluence for (E > 10, > 30, > 60, and > 100 MeV) in table and graphical outputs.

**Free parameters:** None.

<footnote n="8">8 https://cdaw.gsfc.nasa.gov/CME_list/.</footnote>

<footnote n="9">9 http://solardemon.oma.be/flares.php.</footnote>

<footnote n="10">10 https://wwwbis sidc.be/cactus/.</footnote>
**Limitations and caveats:** FORSPEF makes predictions for all ≥C1 flares reported by the Solar DEMON service.

There is a higher level of uncertainty when forecasting flares from ARs near the limbs between 50° to 70° E/W. No forecasts can be made from ARs beyond 70° E/W.

SEP and CME relationships applied in FORSPEF were derived using values from the CDAW CME catalog, which are determined by hand, however the model runs using CME parameters reported in the CACTus catalog and are derived through an automated detection algorithm.

**Access to model output or forecasts:** The FORSPEF system has been online since April 2015 and may be accessed at http://tromos.space.noa.gr/forspef/main/ where it provides continuous forecasts of SFs and SEPs. The system will inform registered users when an SEP event is expected to take place. All forecasts are stored in a fully accessible archived database.

**Model validation:** Anastasiadis et al. (2017) performed a validation of the FORSPEF tool focusing on the POD, FAR, HSS, Overall Accuracy (OA), Critical Success Index (CSI), PCOR, and TSS. For the post-event nowcasting mode, FORSPEF was validated as follows: from the SFs, CMEs and SEPs of SC23 (Papaioannou et al., 2016), 90% of the SFs were selected as the control sample, and the rest (10%) were treated as the test sample, with the selection of the test and control samples being completely random. This procedure was repeated for 10 times and consequently ten different – completely independent – control and test samples were created. For each sample categorical scores were implemented for all test samples with the categorical measures calculated as a function of a Probability Threshold (pt) using SF, CME, and the integrated radio and SXRs flux information as independent inputs. As a result, a pt = 0.25 was identified as the optimal one for the SF and CME modules and a pt = 0.28 for the ESPERTA module. Mean values of selected categorical scores are reported in Table 5, reproduced from Anastasiadis et al. (2017).

Papaioannou et al., 2018 evaluated the performance of the CME nowcasting model using all CMEs from the CDAW catalog between April 2013 to 2015, 43 of which were associated with SEP events recorded by GOES satellite in the >10 MeV energy channel. A probability threshold of 0.3 maximized the HSS derived from the data set. POD, FAR, PCOR, and HSS as a function of probability threshold are plotted in Papaioannou et al. (2018). With a threshold of 0.3, POD is 65% and FAR is 58% for >10 MeV events, POD is 70% and FAR is 15% for

>30 MeV events, and POD is 67% and FAR is 63% for >100 MeV events.

As a next step, Papaioannou et al. (2018) extended the FORSPEF database until the end of 2015, aiming at utilizing the whole time span from the beginning of SC23, i.e., 1997 until 2015. Similar to Anastasiadis et al. (2017), out of the whole sample 85% of all SFs, CMEs, SEPs was chosen as the control sample and the rest 15% as the test sample. This procedure was repeated ten times, and ten different completely independent control and test samples were obtained. For all paired samples (control and test) categorical scores were obtained. The mean POD – out of the 10 different pairs of samples – was 70% and the mean FAR was 44%, which is consistent with the results presented in Anastasiadis et al. (2017).

Papaioannou et al., 2018 assessed the performance of the peak flux prediction using the same set of 43 SEP events. They found significant scatter between the forecasts and observations, but most of the forecasts (≈75%) were within an order of magnitude of the observed value. The corresponding correlation coefficients per energy were $E > 10$ MeV: 0.198 (n = 42), $E > 30$ MeV: 0.235 (n = 33), and $E > 100$ MeV: 0.173 (n = 15).

### 3.10. Georgia State Model

**Model developers:** Berkay Aydin, Manolis K. Georgoulis, Anli Ji, Dustin Kempton, Chetraj Pandey, Rafal Angyk, and Petrus Martens (developed at Georgia State University).

**Model description:** This model is a work in progress, but is intended to become an operational SEP event watch and warning system that runs without human intervention. In its current state, the system provides an All-Clear SEP event forecast, and is composed of a set of connected space weather event forecasting modules, working in different modes. The All-Clear SEP event forecast places emphasis on precise and sensitive prediction of non-flaring (CBN; labeling as 'non-flaring' active regions hosting C-, B- or lower-class flares) active regions in order to identify periods where the occurrence of an SEP event is highly unlikely. In such cases, SEP events of >10 MeV proton flux are not likely to exceed 10 pfu. Later phases of development will implement the SEP Watch, where conditions are likely to lead to an SEP triggering event, and an SEP Warning, where a triggering event has occurred and an SEP event is likely imminent. The connected modules of the system are constructed as ensembles of different learning algo-

<table><caption>Table 5
Mean values of the HSS, POD, FAR, and pt for the ten independent test samples.</caption><thead><tr><td>FORSPEF nowcasting based on</td><td>HSS</td><td>POD</td><td>FAR</td><td>pt</td></tr></thead><tbody><tr><td>Solar flare data</td><td>0.37 ±0.011</td><td>0.40</td><td>0.57</td><td>0.25</td></tr><tr><td>CME data</td><td>0.65 ±0.007</td><td>0.71</td><td>0.41</td><td>0.25</td></tr><tr><td>Radio and SXR fluence data</td><td>0.47 ±0.04</td><td>0.55</td><td>0.42</td><td>0.28</td></tr></tbody></table>
**Limitations and caveats:** FORSPEF makes predictions for all ≥C1 flares reported by the Solar DEMON service.

There is a higher level of uncertainty when forecasting flares from ARs near the limbs between 50° to 70° E/W. No forecasts can be made from ARs beyond 70° E/W.

SEP and CME relationships applied in FORSPEF were derived using values from the CDAW CME catalog, which are determined by hand, however the model runs using CME parameters reported in the CACTus catalog and are derived through an automated detection algorithm.

**Access to model output or forecasts:** The FORSPEF system has been online since April 2015 and may be accessed at http://tromos.space.noa.gr/forspef/main/ where it provides continuous forecasts of SFs and SEPs. The system will inform registered users when an SEP event is expected to take place. All forecasts are stored in a fully accessible archived database.

**Model validation:** Anastasiadis et al. (2017) performed a validation of the FORSPEF tool focusing on the POD, FAR, HSS, Overall Accuracy (OA), Critical Success Index (CSI), PCOR, and TSS. For the post-event nowcasting mode, FORSPEF was validated as follows: from the SFs, CMEs and SEPs of SC23 (Papaioannou et al., 2016), 90% of the SFs were selected as the control sample, and the rest (10%) were treated as the test sample, with the selection of the test and control samples being completely random. This procedure was repeated for 10 times and consequently ten different – completely independent – control and test samples were created. For each sample categorical scores were implemented for all test samples with the categorical measures calculated as a function of a Probability Threshold (pt) using SF, CME, and the integrated radio and SXRs flux information as independent inputs. As a result, a pt = 0.25 was identified as the optimal one for the SF and CME modules and a pt = 0.28 for the ESPERTA module. Mean values of selected categorical scores are reported in Table 5, reproduced from Anastasiadis et al. (2017).

Papaioannou et al., 2018 evaluated the performance of the CME nowcasting model using all CMEs from the CDAW catalog between April 2013 to 2015, 43 of which were associated with SEP events recorded by GOES satellite in the >10 MeV energy channel. A probability threshold of 0.3 maximized the HSS derived from the data set. POD, FAR, PCOR, and HSS as a function of probability threshold are plotted in Papaioannou et al. (2018). With a threshold of 0.3, POD is 65% and FAR is 58% for >10 MeV events, POD is 70% and FAR is 15% for

>30 MeV events, and POD is 67% and FAR is 63% for >100 MeV events.

As a next step, Papaioannou et al. (2018) extended the FORSPEF database until the end of 2015, aiming at utilizing the whole time span from the beginning of SC23, i.e., 1997 until 2015. Similar to Anastasiadis et al. (2017), out of the whole sample 85% of all SFs, CMEs, SEPs was chosen as the control sample and the rest 15% as the test sample. This procedure was repeated ten times, and ten different completely independent control and test samples were obtained. For all paired samples (control and test) categorical scores were obtained. The mean POD – out of the 10 different pairs of samples – was 70% and the mean FAR was 44%, which is consistent with the results presented in Anastasiadis et al. (2017).

Papaioannou et al., 2018 assessed the performance of the peak flux prediction using the same set of 43 SEP events. They found significant scatter between the forecasts and observations, but most of the forecasts (≈75%) were within an order of magnitude of the observed value. The corresponding correlation coefficients per energy were $E > 10$ MeV: 0.198 (n = 42), $E > 30$ MeV: 0.235 (n = 33), and $E > 100$ MeV: 0.173 (n = 15).

### 3.10. Georgia State Model

**Model developers:** Berkay Aydin, Manolis K. Georgoulis, Anli Ji, Dustin Kempton, Chetraj Pandey, Rafal Angyk, and Petrus Martens (developed at Georgia State University).

**Model description:** This model is a work in progress, but is intended to become an operational SEP event watch and warning system that runs without human intervention. In its current state, the system provides an All-Clear SEP event forecast, and is composed of a set of connected space weather event forecasting modules, working in different modes. The All-Clear SEP event forecast places emphasis on precise and sensitive prediction of non-flaring (CBN; labeling as 'non-flaring' active regions hosting C-, B- or lower-class flares) active regions in order to identify periods where the occurrence of an SEP event is highly unlikely. In such cases, SEP events of >10 MeV proton flux are not likely to exceed 10 pfu. Later phases of development will implement the SEP Watch, where conditions are likely to lead to an SEP triggering event, and an SEP Warning, where a triggering event has occurred and an SEP event is likely imminent. The connected modules of the system are constructed as ensembles of different learning algo-

<table><caption>Table 5
Mean values of the HSS, POD, FAR, and pt for the ten independent test samples.</caption><thead><tr><td>FORSPEF nowcasting based on</td><td>HSS</td><td>POD</td><td>FAR</td><td>pt</td></tr></thead><tbody><tr><td>Solar flare data</td><td>0.37 ±0.011</td><td>0.40</td><td>0.57</td><td>0.25</td></tr><tr><td>CME data</td><td>0.65 ±0.007</td><td>0.71</td><td>0.41</td><td>0.25</td></tr><tr><td>Radio and SXR fluence data</td><td>0.47 ±0.04</td><td>0.55</td><td>0.42</td><td>0.28</td></tr></tbody></table>
rithms, producing binary, probabilistic, and regression-based prediction models for flare, eruptive-flare, and CME speed predictors. All predictions are designated for and deployed on individual active regions. All reports are aggregated for a final probabilistic ‘All-Clear’ output based on a set of user-defined thresholds. The All-Clear SEP predictor utilizes Near-Real Time (NRT) HMI Active Region Patch (HARP) data described in Hoeksema et al. (2014) to produce Multivariate Time Series (MVTS) of active-region metadata, with many of the metadata parameter calculation algorithms being the same as the Spaceweather HMI Active Region Patch (SHARP) of Bobra et al. (2014). All models are trained for a 12-h observation window, meaning that a complete prediction result can be issued only after 12 h’ worth of high-quality data has been collected. The prediction window (forecast validity period) for all the models is 24 h with a zero latency (i.e., forecasts are effective immediately). For a discussion of these intervals on flare prediction, see Angryk et al. (2020).

The predictive process follows three distinct paths. First, it determines the probability of a sizable flare (i.e., ≥M1.0) occurrence within the next 24 h. This path uses three base learners (i.e., SOHO-FP, DSDO-FP, NSDO-FP) and a meta learner (which uses the output of base learners), described in Ji et al. (2020), where each base learner is a multivariate Time Series Classifier (TSC) based on the Time Series Forest algorithm from Deng et al. (2013). The second path predicts the probability of an eruptive flare occurrence within the next 24 h. This path also uses the AR MVTS and issues probabilities for occurrence of eruptive (P(ER)) vs. non-eruptive (P(NE)) events. Note here that an eruption may originate from X-, M-, or C-class flares, but not all flares are eruptive. A- and B-class flares are not considered in this framework. The third path uses the outputs of base learners (i.e., the occurrence probability of major flares ≥M1.0) to predict the occurrence probability of X-, M-, C-class flares and Flare-Quiet (FQ) regions. This is a quaternary meta-learner with its outputs fed into a regressor seeking to project a CME speed.

As the prediction algorithms of this system are trained on individual active regions, the results of individual forecasts must then be aggregated. The active region all-clear probability is issued for individual active regions using the following formula:

$$P_{AR-AC} = 1 - [P_{ER}(AR_i) \times \phi_{FL}(P_{FL}(AR_i)) \times \phi_V(V_{CME}(AR_i))]$$

where $P_{ER}$ is the eruption probability, $\phi_{FL}$ is the flare probability activation function, and $\phi_V$ is the CME speed activation function. These activation functions are currently signum functions (thresholded step functions):

$$\phi_{FL} = \begin{cases} 1 & \text{if } P_{FL} > FL_{th} \\ 0 & \text{otherwise} \end{cases} \quad \begin{cases} 1 & \text{if } V_{CME} > V_{th} \\ 0 & \text{otherwise} \end{cases}$$

where the flare probability threshold ($FL_{th}$) and the CME speed threshold ($V_{th}$) are set to a user defined values. The full disk all clear probability ($P_{AC}$) is then calculated as the joint all-clear probability of active region all-clear output. The probabilities of eruptions from active regions are assumed to be statistically independent, making the joint probability a simple product of the observed active regions at the time of model report.

**Inputs:** The current model input consists of the NRT data series of the HARP data product described in Hoeksema et al. (2014). The most recent observations are collected from the source repository at the Joint Science Operations Center (http://jsoc.stanford.edu/) for each active region recorded at the report time. The data for each active region is collected at 12 min intervals over the previous 12 h.

**Outputs:**

* The probability of occurrence for the peak proton flux exceeding 10 pfu for proton energies $E \ge 10$ MeV over the next 24 h

* A binary all clear flag indicating that the probability of peak proton flux exceeding 10 pfu has exceeded some threshold (currently 25%)

**Free parameters:** This is a combination of several machine learning models and it does not contain free parameters in the same sense as a physical model. However, when deploying the system there are several parameters that can be set to change runtime frequency, as well as assumptions about weighting of intermediary results and thresholds for consideration in calculations.

* Solar maximum climatological eruptive flaring rate for any given active region observation time: used when data for an active region is unreliable and the model is set to assume it is running during solar maximum;

* Solar minimum climatological eruptive flaring rate for any given active region observation time: used when data for an active region is unreliable and the model is set to assume it is running during solar minimum;

* CME Speed Threshold is used to determine when an active region will be included in the All-Clear calculation. The active regions predicted by the CME speed module not to produce a CME exceeding this speed will be assumed to be clear from possibly producing an SEP event and will be excluded from calculation;

* M and X flare Threshold is used to determine when an active region will be included in the All-Clear calculation. The active regions that the flare prediction meta model output of Ji et al. (2020) does not predict will produce an M or X class flare with this probability threshold are excluded from calculation.

**Limitations and caveats:** The All-Clear forecasting heuristic makes two major assumptions to provide conservative All-Clear forecasts: first, it assumes that all predicted
rithms, producing binary, probabilistic, and regression-based prediction models for flare, eruptive-flare, and CME speed predictors. All predictions are designated for and deployed on individual active regions. All reports are aggregated for a final probabilistic ‘All-Clear’ output based on a set of user-defined thresholds. The All-Clear SEP predictor utilizes Near-Real Time (NRT) HMI Active Region Patch (HARP) data described in Hoeksema et al. (2014) to produce Multivariate Time Series (MVTS) of active-region metadata, with many of the metadata parameter calculation algorithms being the same as the Spaceweather HMI Active Region Patch (SHARP) of Bobra et al. (2014). All models are trained for a 12-h observation window, meaning that a complete prediction result can be issued only after 12 h’ worth of high-quality data has been collected. The prediction window (forecast validity period) for all the models is 24 h with a zero latency (i.e., forecasts are effective immediately). For a discussion of these intervals on flare prediction, see Angryk et al. (2020).

The predictive process follows three distinct paths. First, it determines the probability of a sizable flare (i.e., ≥M1.0) occurrence within the next 24 h. This path uses three base learners (i.e., SOHO-FP, DSDO-FP, NSDO-FP) and a meta learner (which uses the output of base learners), described in Ji et al. (2020), where each base learner is a multivariate Time Series Classifier (TSC) based on the Time Series Forest algorithm from Deng et al. (2013). The second path predicts the probability of an eruptive flare occurrence within the next 24 h. This path also uses the AR MVTS and issues probabilities for occurrence of eruptive (P(ER)) vs. non-eruptive (P(NE)) events. Note here that an eruption may originate from X-, M-, or C-class flares, but not all flares are eruptive. A- and B-class flares are not considered in this framework. The third path uses the outputs of base learners (i.e., the occurrence probability of major flares ≥M1.0) to predict the occurrence probability of X-, M-, C-class flares and Flare-Quiet (FQ) regions. This is a quaternary meta-learner with its outputs fed into a regressor seeking to project a CME speed.

As the prediction algorithms of this system are trained on individual active regions, the results of individual forecasts must then be aggregated. The active region all-clear probability is issued for individual active regions using the following formula:

$$P_{AR-AC} = 1 - [P_{ER}(AR_i) \times \phi_{FL}(P_{FL}(AR_i)) \times \phi_V(V_{CME}(AR_i))]$$

where $P_{ER}$ is the eruption probability, $\phi_{FL}$ is the flare probability activation function, and $\phi_V$ is the CME speed activation function. These activation functions are currently signum functions (thresholded step functions):

$$\phi_{FL} = \begin{cases} 1 & \text{if } P_{FL} > FL_{th} \\ 0 & \text{otherwise} \end{cases} \quad \begin{cases} 1 & \text{if } V_{CME} > V_{th} \\ 0 & \text{otherwise} \end{cases}$$

where the flare probability threshold ($FL_{th}$) and the CME speed threshold ($V_{th}$) are set to a user defined values. The full disk all clear probability ($P_{AC}$) is then calculated as the joint all-clear probability of active region all-clear output. The probabilities of eruptions from active regions are assumed to be statistically independent, making the joint probability a simple product of the observed active regions at the time of model report.

**Inputs:** The current model input consists of the NRT data series of the HARP data product described in Hoeksema et al. (2014). The most recent observations are collected from the source repository at the Joint Science Operations Center (http://jsoc.stanford.edu/) for each active region recorded at the report time. The data for each active region is collected at 12 min intervals over the previous 12 h.

**Outputs:**

* The probability of occurrence for the peak proton flux exceeding 10 pfu for proton energies $E \ge 10$ MeV over the next 24 h

* A binary all clear flag indicating that the probability of peak proton flux exceeding 10 pfu has exceeded some threshold (currently 25%)

**Free parameters:** This is a combination of several machine learning models and it does not contain free parameters in the same sense as a physical model. However, when deploying the system there are several parameters that can be set to change runtime frequency, as well as assumptions about weighting of intermediary results and thresholds for consideration in calculations.

* Solar maximum climatological eruptive flaring rate for any given active region observation time: used when data for an active region is unreliable and the model is set to assume it is running during solar maximum;

* Solar minimum climatological eruptive flaring rate for any given active region observation time: used when data for an active region is unreliable and the model is set to assume it is running during solar minimum;

* CME Speed Threshold is used to determine when an active region will be included in the All-Clear calculation. The active regions predicted by the CME speed module not to produce a CME exceeding this speed will be assumed to be clear from possibly producing an SEP event and will be excluded from calculation;

* M and X flare Threshold is used to determine when an active region will be included in the All-Clear calculation. The active regions that the flare prediction meta model output of Ji et al. (2020) does not predict will produce an M or X class flare with this probability threshold are excluded from calculation.

**Limitations and caveats:** The All-Clear forecasting heuristic makes two major assumptions to provide conservative All-Clear forecasts: first, it assumes that all predicted
eruptions (CMEs) are directed towards Earth face-on (i.e., they are halo CMEs) and, second, it assumes that all of them can potentially lead to an SEP event. Therefore, if either of these events is expected, an All-Clear forecast will not be issued.

**Access to model output or forecasts:** Model forecasts can be accessed at https://dmlab.cs.gsu.edu/sep-prediction/.

**Model validation:** The various sub-models of the system were trained and tested with a set time segmented partitions to enable cross-validation. For instance, the SOHO-FP flare prediction sub-model utilized a SOHO dataset much like the HARP dataset, divided into three partitions allowing a 2/3 training vs. 1/3 testing split for each iteration. As discussed in Ji et al. (2020), training and testing on the DSDO-FP sub-model utilizes 5 partitions from a machine learning ready dataset for the flare prediction task described in Angryk et al. (2020). Similarly, the NSDO-FP sub-module utilizes the same partitioning strategy on the NRT HARP data to construct the partitions, instead of the definitive data used in Angryk et al. (2020).

### 3.11. iPATH - improved Particle Acceleration and Transport in the Heliosphere

**Model developers:** Gang Li, Junxiang Hu, Gary Zank (University of Alabama in Huntsville).

**Model description:** The improved Particle Acceleration and Transport in the Heliosphere (IPATH) model is a 2-D MHD SEP model that simulates Diffusive Shock Acceleration (DSA) at CME-driven shocks and follows the subsequent transport of energetic particles through the inner heliosphere. IPATH models the background solar wind and CME-driven shocks at the ecliptic plane starting at 0.05 AU and produces time profiles of SEP intensity spectra and pitch angle distributions as outputs at selected voltage points (e.g., at Earth or Mars). It considers both perpendicular and parallel diffusive factors of energetic particles, which come from Nonlinear Guiding Center Theory (NLGC) and Quasi-Linear Theory (QLT) respectively. The transport module is a Monte Carlo code which follows test particles through space described by the FTE and is set up for parallel computations. This model is improved over the original PATH model which was 1-D.

IPATH first creates the shock perturbation along the inner boundary, and propagates the CME outward with the forward shock region tracked with a 2-D onion-shell model. For each time step, a new outer shell is created based on the shock speed and all previous shells convect and adiabatically expand with the solar wind. Then accelerated particle distributions are calculated along the whole shock front, based on the diffusive shock acceleration. Accelerated particles are then allowed to diffuse back to the shock complex, and between each parcel behind the shock via parallel and perpendicular diffusion. This gives the distribution function in each shock parcel at each time step, which is important for the ESP phase when the shock

arrives at the observer. Once a particle has moved a certain distance during a single time step, it escapes the shock and is transported through the unperturbed solar wind via a focused transport scheme, which includes terms for weak scattering and cross-field diffusion from the random walk of magnetic field lines. The FTE is solved using a time-backwards stochastic differential equation approach until a steady-state is found where ensemble averages of many test particle paths give the full particle distribution function. The full model description is detailed in Hu et al. (2017).

**Inputs:**

* Background solar wind parameters based on 1 AU observations (solar wind density, speed, temperature, and magnetic field strength)

* CME parameters (CME speed, width, location, and perturbation duration)

* Turbulence parameters (turbulence level, turbulence spectral parameters, and radial dependencies of these parameters)

* Suprathermal seed particle input (suprathermal particle energy spectral parameters)

* Observer locations (radius, longitude)

**Outputs:**

* Time profiles for energetic proton/heavy ion intensities (differential fluxes) at a wide energy range (from hundreds of keV's to GeVs) at the chosen observer locations

* Time profiles for pitch angle distributions

* Event-integrated fluences (in $MeV^{-1}cm^{-2}$)

* Time profiles for integral flux above certain energies (in pfu)

**Free parameters:** None.

**Limitations and caveats:** IPATH as a coupled MHD and acceleration/transport model has many advantages at modeling the underlying physics during a SEP event. For instance, its inclusion of perpendicular diffusion allows for a large longitudinal spread of energetic particles. Thus, it is well suited to model events that are not well-connected from the CME to the observer. It can also generate results at multiple observer locations throughout the inner heliosphere. Using the latest developments in the modeling of seed particles, IPATH is able to approximate the solar wind's background pre-event particles that are to be injected into the DSA. Additionally, as distribution functions of particles trapped in the shock downstream are also tracked, IPATH is capable of modeling the ESP phase.

Like other models using CME information as inputs, IPATH's performance as a prediction model relies on how early we can acquire good quality CME parameters from observations. As the inner boundary is set at 0.05 AU (10 $R_{sun}$) and the shock is formed even further in, it cannot capture the SEP acceleration happening in the low corona and does not produce output for the first
eruptions (CMEs) are directed towards Earth face-on (i.e., they are halo CMEs) and, second, it assumes that all of them can potentially lead to an SEP event. Therefore, if either of these events is expected, an All-Clear forecast will not be issued.

**Access to model output or forecasts:** Model forecasts can be accessed at https://dmlab.cs.gsu.edu/sep-prediction/.

**Model validation:** The various sub-models of the system were trained and tested with a set time segmented partitions to enable cross-validation. For instance, the SOHO-FP flare prediction sub-model utilized a SOHO dataset much like the HARP dataset, divided into three partitions allowing a 2/3 training vs. 1/3 testing split for each iteration. As discussed in Ji et al. (2020), training and testing on the DSDO-FP sub-model utilizes 5 partitions from a machine learning ready dataset for the flare prediction task described in Angryk et al. (2020). Similarly, the NSDO-FP sub-module utilizes the same partitioning strategy on the NRT HARP data to construct the partitions, instead of the definitive data used in Angryk et al. (2020).

### 3.11. iPATH - improved Particle Acceleration and Transport in the Heliosphere

**Model developers:** Gang Li, Junxiang Hu, Gary Zank (University of Alabama in Huntsville).

**Model description:** The improved Particle Acceleration and Transport in the Heliosphere (IPATH) model is a 2-D MHD SEP model that simulates Diffusive Shock Acceleration (DSA) at CME-driven shocks and follows the subsequent transport of energetic particles through the inner heliosphere. IPATH models the background solar wind and CME-driven shocks at the ecliptic plane starting at 0.05 AU and produces time profiles of SEP intensity spectra and pitch angle distributions as outputs at selected voltage points (e.g., at Earth or Mars). It considers both perpendicular and parallel diffusive factors of energetic particles, which come from Nonlinear Guiding Center Theory (NLGC) and Quasi-Linear Theory (QLT) respectively. The transport module is a Monte Carlo code which follows test particles through space described by the FTE and is set up for parallel computations. This model is improved over the original PATH model which was 1-D.

IPATH first creates the shock perturbation along the inner boundary, and propagates the CME outward with the forward shock region tracked with a 2-D onion-shell model. For each time step, a new outer shell is created based on the shock speed and all previous shells convect and adiabatically expand with the solar wind. Then accelerated particle distributions are calculated along the whole shock front, based on the diffusive shock acceleration. Accelerated particles are then allowed to diffuse back to the shock complex, and between each parcel behind the shock via parallel and perpendicular diffusion. This gives the distribution function in each shock parcel at each time step, which is important for the ESP phase when the shock

arrives at the observer. Once a particle has moved a certain distance during a single time step, it escapes the shock and is transported through the unperturbed solar wind via a focused transport scheme, which includes terms for weak scattering and cross-field diffusion from the random walk of magnetic field lines. The FTE is solved using a time-backwards stochastic differential equation approach until a steady-state is found where ensemble averages of many test particle paths give the full particle distribution function. The full model description is detailed in Hu et al. (2017).

**Inputs:**

* Background solar wind parameters based on 1 AU observations (solar wind density, speed, temperature, and magnetic field strength)

* CME parameters (CME speed, width, location, and perturbation duration)

* Turbulence parameters (turbulence level, turbulence spectral parameters, and radial dependencies of these parameters)

* Suprathermal seed particle input (suprathermal particle energy spectral parameters)

* Observer locations (radius, longitude)

**Outputs:**

* Time profiles for energetic proton/heavy ion intensities (differential fluxes) at a wide energy range (from hundreds of keV's to GeVs) at the chosen observer locations

* Time profiles for pitch angle distributions

* Event-integrated fluences (in $MeV^{-1}cm^{-2}$)

* Time profiles for integral flux above certain energies (in pfu)

**Free parameters:** None.

**Limitations and caveats:** IPATH as a coupled MHD and acceleration/transport model has many advantages at modeling the underlying physics during a SEP event. For instance, its inclusion of perpendicular diffusion allows for a large longitudinal spread of energetic particles. Thus, it is well suited to model events that are not well-connected from the CME to the observer. It can also generate results at multiple observer locations throughout the inner heliosphere. Using the latest developments in the modeling of seed particles, IPATH is able to approximate the solar wind's background pre-event particles that are to be injected into the DSA. Additionally, as distribution functions of particles trapped in the shock downstream are also tracked, IPATH is capable of modeling the ESP phase.

Like other models using CME information as inputs, IPATH's performance as a prediction model relies on how early we can acquire good quality CME parameters from observations. As the inner boundary is set at 0.05 AU (10 $R_{sun}$) and the shock is formed even further in, it cannot capture the SEP acceleration happening in the low corona and does not produce output for the first
couple of hours of the events. IPATH is not able to fully
capture the effects of previous CMEs without modifications
on the model so the operational version is best suited for
single CME events. The background solar wind used in
IPATH is a simple homogeneous Parker spiral model,
which cannot capture some complex solar wind geometries
in a specific event. But this simplification is efficient for
general prediction purposes.

**Access to model output or forecasts:** iPATH is in the pro-
cess of implementing the model at CCMC for Runs on
Request and the developers are in the process of developing
a real-time forecasting module for the SEP Scoreboards.

**Model validation:** Originally IPATH was only used as a research model for historic SEP events (see Ding et al., 2020 as an example), so no large validation study has been done with the model. More recently, IPATH has been involved with the International Space Weather Action Teams (ISWAT)/Solar Heliospheric and INterplanetary Environment workshop (SHINE) community SEP model validation challenge.

The next steps for IPATH include:

* Coupling with a coronal model to account for acceleration before the shock reaches 0.1 AU - see coupling of IPATH and AWSOM called SEPCaster (Section 3.24).

* Implementation of a nowcasting mode using real time solar wind inputs and near continuous runs for real events, as part of the SEP Scoreboards.

3.12. Lavasa Model

**Model developers:** Eleni Lavasa (National and Kapodistrian University of Athens) and Giorgos Giannopoulos (Information Management Systems Institute (IMSI) - "Athena" Research Center).

**Model description:** A consistent ML framework for the binary prediction of SEP events, based on SF and CME observational data has been implemented. A thorough investigation for optimal solutions was performed under our proposed approach, across diverse classification algorithms and extended hyper-parameter spaces. A thorough experimental evaluation, with the twofold purpose to uncover informative combinations of the selected SXR flare and/or CME input variables, as well as to establish the proper setting for the inherently imbalanced problem at hand was performed. Random Forest (RF), a tree-based ensemble model is found to be the best performing and more robust solution in our evaluation setting, running on both flare and CME data to achieve a POD = 0.76 (±0.06), a FAR = 0.34(±0.10), TSS = 0.75(±0.05) and HSS = 0.69(±0.04). Post-hoc analysis was performed, to relate the recovered ML models to the underlying physics of SEP drivers. CME speed and width, along with flare SXR fluence are highlighted as the features with the strongest discriminatory capabilities in our setting.

The prediction of SEP events was treated as a super-
vised binary classification task. Solar eruptive events (i.e.
flares, CMEs and their combination) can be allocated to
one of two possible classes, depending on whether these
are associated with an SEP event (>10 MeV) or not.
Thereby, two classes are defined (i.e., 0/1 – no/yes).
The models that were constructed include: Logistic
Regression (LR), Support Vector Machines with Linear
kernels (linSVM) and Non-Linear kernels (SVM), NN,
RF, Extremely Randomized Trees (ET) and Extreme
Gradient Boosting (XGB) (see details in Lavasa et al.,
2021).

**Inputs:**

Depend on the dataset used

Characteristics of solar flares and CMEs features were
extracted from the source database (Papaioannou et al.,
2016), in order to construct the input variable space to
all models. These features include:

i Solar-flare heliographic longitude in degrees

ii The log10 value of flare peak intensity (W/m2) in
SXR (1 – 8 Å)

iii The log10 value of SXR flare time-integrated flux in J/m²

iv Duration of SXR flare in minutes

v Rise time of SXR flare in minutes

vi CME apparent angular width in the sky-plane projection in degrees

vii CME linear speed in km/s

viii Cycle index was created as a simple numerical fea-
ture, indicative to the magnitude of solar activity at
the time when an event occurred

Then *datasets* were created as follows:

1. **Flare dataset** (includes i-v & viii features)

2. CME dataset (includes vi-viii features)

3. **Flare & CME dataset** (includes all i-viii features)

In addition, these datasets were treated as *Imbalanced* and *Balanced* in two parallel experimental settings, depending on whether all (i.e., negative and positive) class instances were included, or a subset of negative class instances was kept via undersampling, so that the dataset contained an equal number of positive and negative instances. Finally, another condition was imposed based on the treatment applied to Missing Value (MV) s. Thereby, *Noiseless* datasets are the ones where all entries having at least one MV were eliminated and *Noisy* datasets are those for which MVs were filled with the feature's median value.

**Outputs:**

Binary classification includes SEP-associated eruptive
events that are assigned a label of + 1, hence the positive
class, while the negative class includes eruptive events with-
out SEP association that are assigned a label of −1. All of
couple of hours of the events. IPATH is not able to fully
capture the effects of previous CMEs without modifications
on the model so the operational version is best suited for
single CME events. The background solar wind used in
IPATH is a simple homogeneous Parker spiral model,
which cannot capture some complex solar wind geometries
in a specific event. But this simplification is efficient for
general prediction purposes.

**Access to model output or forecasts:** iPATH is in the pro-
cess of implementing the model at CCMC for Runs on
Request and the developers are in the process of developing
a real-time forecasting module for the SEP Scoreboards.

**Model validation:** Originally IPATH was only used as a research model for historic SEP events (see Ding et al., 2020 as an example), so no large validation study has been done with the model. More recently, IPATH has been involved with the International Space Weather Action Teams (ISWAT)/Solar Heliospheric and INterplanetary Environment workshop (SHINE) community SEP model validation challenge.

The next steps for IPATH include:

* Coupling with a coronal model to account for acceleration before the shock reaches 0.1 AU - see coupling of IPATH and AWSOM called SEPCaster (Section 3.24).

* Implementation of a nowcasting mode using real time solar wind inputs and near continuous runs for real events, as part of the SEP Scoreboards.

3.12. Lavasa Model

**Model developers:** Eleni Lavasa (National and Kapodistrian University of Athens) and Giorgos Giannopoulos (Information Management Systems Institute (IMSI) - "Athena" Research Center).

**Model description:** A consistent ML framework for the binary prediction of SEP events, based on SF and CME observational data has been implemented. A thorough investigation for optimal solutions was performed under our proposed approach, across diverse classification algorithms and extended hyper-parameter spaces. A thorough experimental evaluation, with the twofold purpose to uncover informative combinations of the selected SXR flare and/or CME input variables, as well as to establish the proper setting for the inherently imbalanced problem at hand was performed. Random Forest (RF), a tree-based ensemble model is found to be the best performing and more robust solution in our evaluation setting, running on both flare and CME data to achieve a POD = 0.76 (±0.06), a FAR = 0.34(±0.10), TSS = 0.75(±0.05) and HSS = 0.69(±0.04). Post-hoc analysis was performed, to relate the recovered ML models to the underlying physics of SEP drivers. CME speed and width, along with flare SXR fluence are highlighted as the features with the strongest discriminatory capabilities in our setting.

The prediction of SEP events was treated as a super-
vised binary classification task. Solar eruptive events (i.e.
flares, CMEs and their combination) can be allocated to
one of two possible classes, depending on whether these
are associated with an SEP event (>10 MeV) or not.
Thereby, two classes are defined (i.e., 0/1 – no/yes).
The models that were constructed include: Logistic
Regression (LR), Support Vector Machines with Linear
kernels (linSVM) and Non-Linear kernels (SVM), NN,
RF, Extremely Randomized Trees (ET) and Extreme
Gradient Boosting (XGB) (see details in Lavasa et al.,
2021).

Inputs:

Depend on the dataset used

Characteristics of solar flares and CMEs features were
extracted from the source database (Papaioannou et al.,
2016), in order to construct the input variable space to
all models. These features include:

i Solar-flare heliographic longitude in degrees

ii The log10 value of flare peak intensity (W/m2) in
SXR (1 − 8 Å)

iii The log10 value of SXR flare time-integrated flux in J/m²

iv Duration of SXR flare in minutes

v Rise time of SXR flare in minutes

vi CME apparent angular width in the sky-plane projection in degrees

vii CME linear speed in km/s

viii Cycle index was created as a simple numerical fea-
ture, indicative to the magnitude of solar activity at
the time when an event occurred

Then *datasets* were created as follows:

1. **Flare dataset** (includes i-v & viii features)

2. CME dataset (includes vi-viii features)

3. **Flare & CME dataset** (includes all i-viii features)

In addition, these datasets were treated as *Imbalanced* and *Balanced* in two parallel experimental settings, depending on whether all (i.e., negative and positive) class instances were included, or a subset of negative class instances was kept via undersampling, so that the dataset contained an equal number of positive and negative instances. Finally, another condition was imposed based on the treatment applied to Missing Value (MV) s. Thereby, *Noiseless* datasets are the ones where all entries having at least one MV were eliminated and *Noisy* datasets are those for which MVs were filled with the feature's median value.

Outputs:

Binary classification includes SEP-associated eruptive
events that are assigned a label of + 1, hence the positive
class, while the negative class includes eruptive events with-
out SEP association that are assigned a label of −1. All of
the above models (see SubSection 3.12) provided outputs for all of the created datasets (see SubSection 3.12).

**Free parameters:**

The free parameters (hyper-parameters) for each of the examined ML models, are tuned in the inner loop of the adopted nested cross-validation scheme. In this process, each model is built for 1000 randomly chosen hyper-parameter configurations within an extended hyper-parameter grid. The optimal configuration is selected based on mean validation scores. For the best-performing and more robust amongst the examined models, RF, optimized with respect to Precision (=1-FAR), the optimal hyper-parameter set recovered in all five folds is:

* Number of estimators = 800

* Split criterion = Entropy

* Minimum samples per split = 5

* Minimum samples per leaf = 2

* Maximum depth = 50

* Maximum number of features = sqrt(number of features)

* Class weight = 0: 1, 1:500

We note however that different hyper-parameter configurations may be recovered in each fold, but still provide consistent predictions, as is the case when we optimize

RF by means of the F1-score. In any case, our proposed pipeline recovers the optimal free parameter configurations in each fold, to evaluate the combined performance of (classification model + hyper-parameter grid) on the given data.

**Limitations and caveats:** All models predict the occurrence (or not) of an upcoming SEP event.

**Access to model output or forecasts:** None of the implemented ML models have been set into operation. It is envisioned, however, that a selection of the implemented models will be integrated as an independent module into the ASPECS tool in order to facilitate direct comparison of the predictions in archived and near real-time data.

**Model validation:** All models were validated using probabilistic metrics (i.e., POD, FAR, TSS, HSS, F1-Score) for both the established *Imbalanced* and *Balanced* evaluation settings. The reported results in Lavasa et al., 2021 are calculated on the unseen, test partitions of a nested cross-validation scheme, averaged across folds. Out of the datasets configured and employed, the examined models achieve higher SEP predictability effectiveness, in the real-world *Imbalanced* setting, when applied on the *Flare_CME_Noiseless* one. As it can be seen in Fig. 3, RF, a tree-based ensemble model is the best performing and more robust solution in our evaluation setting, running on both flare and CME data, achieving POD = 0.76

<figure><img src="image_4.png" /><figcaption>Fig. 3. Lavasa Models’ performance plots: POD, FAR, TSS and HSS, on the *Flare_CME_Noiseless* dataset configuration. Results obtained by the *Imbalanced* evaluation settings under F1 optimization are displayed. The scores are constructed for each of the eight models employed, which are printed on the x-axis. The mean and median values are indicated by filled triangles and thick horizontal black lines, respectively, while gray diamonds indicate outliers. Figure adapted by Lavasa et al. (2021).</figcaption></figure>
the above models (see SubSection 3.12) provided outputs for all of the created datasets (see SubSection 3.12).

**Free parameters:**

The free parameters (hyper-parameters) for each of the examined ML models, are tuned in the inner loop of the adopted nested cross-validation scheme. In this process, each model is built for 1000 randomly chosen hyper-parameter configurations within an extended hyper-parameter grid. The optimal configuration is selected based on mean validation scores. For the best-performing and more robust amongst the examined models, RF, optimized with respect to Precision (=1-FAR), the optimal hyper-parameter set recovered in all five folds is:

* Number of estimators = 800

* Split criterion = Entropy

* Minimum samples per split = 5

* Minimum samples per leaf = 2

* Maximum depth = 50

* Maximum number of features = sqrt(number of features)

* Class weight = 0: 1, 1:500

We note however that different hyper-parameter configurations may be recovered in each fold, but still provide consistent predictions, as is the case when we optimize

RF by means of the F1-score. In any case, our proposed pipeline recovers the optimal free parameter configurations in each fold, to evaluate the combined performance of (classification model + hyper-parameter grid) on the given data.

**Limitations and caveats:** All models predict the occurrence (or not) of an upcoming SEP event.

**Access to model output or forecasts:** None of the implemented ML models have been set into operation. It is envisioned, however, that a selection of the implemented models will be integrated as an independent module into the ASPECS tool in order to facilitate direct comparison of the predictions in archived and near real-time data.

**Model validation:** All models were validated using probabilistic metrics (i.e., POD, FAR, TSS, HSS, F1-Score) for both the established *Imbalanced* and *Balanced* evaluation settings. The reported results in Lavasa et al., 2021 are calculated on the unseen, test partitions of a nested cross-validation scheme, averaged across folds. Out of the datasets configured and employed, the examined models achieve higher SEP predictability effectiveness, in the real-world *Imbalanced* setting, when applied on the *Flare_CME_Noiseless* one. As it can be seen in Fig. 3, RF, a tree-based ensemble model is the best performing and more robust solution in our evaluation setting, running on both flare and CME data, achieving POD = 0.76

<figure><img src="image_4.png" /><figcaption>Fig. 3. Lavasa Models’ performance plots: POD, FAR, TSS and HSS, on the *Flare_CME_Noiseless* dataset configuration. Results obtained by the *Imbalanced* evaluation settings under F1 optimization are displayed. The scores are constructed for each of the eight models employed, which are printed on the x-axis. The mean and median values are indicated by filled triangles and thick horizontal black lines, respectively, while gray diamonds indicate outliers. Figure adapted by Lavasa et al. (2021).</figcaption></figure>
(±0.06), FAR = 0.34(±0.10), TSS = 0.75(±0.05) and HSS = 0.69(±0.04). As a next step in the evaluation of the ML models the most important features employed in our study that, in turn, would signify the highest prediction capability, were searched for. The average permutation importance scores were calculated, for all models utilizing both flare and CME features in the *Flare_CME_Noiseless* dataset signifying that the CME speed, width and SXR fluence are the prime proxies for the identification of an SEP event.

### 3.13. MAG4 - Magnetogram Forecast

**Model developers:** David Falconer (University of Alabama in Huntsville).

**Model description:** Magnetogram Forecast (MAG4) is a probabilistic forecasting tool that is capable of giving probabilities of the following events: M + X-class flares, X-class flares, CMEs, fast CMEs, and SPEs. MAG4 is suited to operational use because it forecasts the probability of >10 MeV protons crossing the 10 Particle Flux Units (1/cm² -s-ster) (pfu) threshold prior to observation of the actual event. The model is fully automated, runs in real-time, and is widely used within the forecasting and science communities.

MAG4 starts with magnetograms from SDO/HMI in the form of SHARP data. SHARP data includes an algorithm that identifies an AR on the solar disk. Depending on the forecast mode in MAG4, the model makes use of either the line-of-sight (LOS) or vector magnetograms. Additionally, if Michelson Doppler Imager (MDI) forecast curves are being used, the HMI magnetograms are smoothed to the (lower) MDI resolution.

For each SHARP, MAG4 performs the following steps:

* Use an analytic solution (Alissandrakis, 1981) to calculate the horizontal potential field from the vertical magnetic field at every pixel. In the LOS mode, MAG4 uses the LOS magnetic field to calculate the transverse potential field.

* Use a contour scheme to identify the neutral line. Apply a threshold to the neutral line such that the transverse potential field must be greater than 150 G to capture the neutral line between strong magnetic fields.

* Calculate the following magnetic parameters:

- Length of the strong-field neutral line.

- Area with $|B_z| > 100$ G.

- Magnetic flux with $|B_z| > 100$ G.

- Shear angle between the magnetic field and transverse potential field across each pixel on the neutral line.

- Gradient of the magnetic field across each pixel on the neutral line.

* Remove any ARs where the length of the neutral line divided by the square root of the area with $|B_z| > 100$ G is less than 0.75.

* Use the shear angle and gradient along the neutral line to get the event rate per day based on previously derived forecast curves for each event type.

* Use Poisson statistics to convert the event rate per day to a probability. This is done while including the event rates across all ARs currently on the disk.

**Inputs:**

* Magnetograms from one or both of the following instruments:

- SOHO/ MDI

- SDO/ HMI

- Global Oscillation Network Group (GONG) (as a back-up source)

* “MAG4 w/ flaring” model sub-type includes M- and X-class flare history from the previous 24 h for flare productive ARs.

**Outputs:** Probabilities of occurrence of the following:

* M + X-class flares

* X-class flares

* CMEs

* Fast CMEs

* SPEs

MAG4 provides a graphic identifying the locations of each AR identified by the model overlayed on a magnetogram image. A graphical “Threat Gauge” shows the cumulative probability of occurrence for each of the phenomena in the list above.

**Free parameters:** None.

**Limitations and caveats:** MAG4 forecasts are most accurate for ARs within a 45-degree cone from disk center.

**Access to model output or forecasts:** Model is available at CCMC for Runs on Request, as well as on the SEP Probability Scoreboard. Model results can also be viewed on MAG4's website: https://www.uah.edu/cspar/research/mag4-page.

**Model validation:** Model validation has been performed for a version of MAG4 that uses HMI data smoothed to MDI resolution, MDI forecast curves, and previous flare activity Falconer et al. (2014). The study used a montecarlo technique similar to k-fold cross validation. A fraction of the data set was set aside for testing. The remaining data set was used to train the model, then tested against the training data set. This was repeated many times, and an average skill score was calculated. The results are shown in Table 6 and include a comparison to the model without using previous flare activity.

Space Radiation Analysis Group (SRAG) provided a list of >100 MeV, >1 pfu events and >50 MeV, >0.5 pfu events for verification purposes. Filament eruption events were removed from the event list. Also eliminated from the event list were events with >10 MeV, >10 pfu that
(±0.06), FAR = 0.34(±0.10), TSS = 0.75(±0.05) and HSS = 0.69(±0.04). As a next step in the evaluation of the ML models the most important features employed in our study that, in turn, would signify the highest prediction capability, were searched for. The average permutation importance scores were calculated, for all models utilizing both flare and CME features in the *Flare_CME_Noiseless* dataset signifying that the CME speed, width and SXR fluence are the prime proxies for the identification of an SEP event.

### 3.13. MAG4 - Magnetogram Forecast

**Model developers:** David Falconer (University of Alabama in Huntsville).

**Model description:** Magnetogram Forecast (MAG4) is a probabilistic forecasting tool that is capable of giving probabilities of the following events: M + X-class flares, X-class flares, CMEs, fast CMEs, and SPEs. MAG4 is suited to operational use because it forecasts the probability of >10 MeV protons crossing the 10 Particle Flux Units (1/cm² -s-ster) (pfu) threshold prior to observation of the actual event. The model is fully automated, runs in real-time, and is widely used within the forecasting and science communities.

MAG4 starts with magnetograms from SDO/HMI in the form of SHARP data. SHARP data includes an algorithm that identifies an AR on the solar disk. Depending on the forecast mode in MAG4, the model makes use of either the line-of-sight (LOS) or vector magnetograms. Additionally, if Michelson Doppler Imager (MDI) forecast curves are being used, the HMI magnetograms are smoothed to the (lower) MDI resolution.

For each SHARP, MAG4 performs the following steps:

* Use an analytic solution (Alissandrakis, 1981) to calculate the horizontal potential field from the vertical magnetic field at every pixel. In the LOS mode, MAG4 uses the LOS magnetic field to calculate the transverse potential field.

* Use a contour scheme to identify the neutral line. Apply a threshold to the neutral line such that the transverse potential field must be greater than 150 G to capture the neutral line between strong magnetic fields.

* Calculate the following magnetic parameters:

- Length of the strong-field neutral line.

- Area with $|B_z| > 100$ G.

- Magnetic flux with $|B_z| > 100$ G.

- Shear angle between the magnetic field and transverse potential field across each pixel on the neutral line.

- Gradient of the magnetic field across each pixel on the neutral line.

* Remove any ARs where the length of the neutral line divided by the square root of the area with $|B_z| > 100$ G is less than 0.75.

* Use the shear angle and gradient along the neutral line to get the event rate per day based on previously derived forecast curves for each event type.

* Use Poisson statistics to convert the event rate per day to a probability. This is done while including the event rates across all ARs currently on the disk.

**Inputs:**

* Magnetograms from one or both of the following instruments:

- SOHO/ MDI

- SDO/ HMI

- Global Oscillation Network Group (GONG) (as a back-up source)

* “MAG4 w/ flaring” model sub-type includes M- and X-class flare history from the previous 24 h for flare productive ARs.

**Outputs:** Probabilities of occurrence of the following:

* M + X-class flares

* X-class flares

* CMEs

* Fast CMEs

* SPEs

MAG4 provides a graphic identifying the locations of each AR identified by the model overlayed on a magnetogram image. A graphical “Threat Gauge” shows the cumulative probability of occurrence for each of the phenomena in the list above.

**Free parameters:** None.

**Limitations and caveats:** MAG4 forecasts are most accurate for ARs within a 45-degree cone from disk center.

**Access to model output or forecasts:** Model is available at CCMC for Runs on Request, as well as on the SEP Probability Scoreboard. Model results can also be viewed on MAG4's website: https://www.uah.edu/cspar/research/mag4-page.

**Model validation:** Model validation has been performed for a version of MAG4 that uses HMI data smoothed to MDI resolution, MDI forecast curves, and previous flare activity Falconer et al. (2014). The study used a montecarlo technique similar to k-fold cross validation. A fraction of the data set was set aside for testing. The remaining data set was used to train the model, then tested against the training data set. This was repeated many times, and an average skill score was calculated. The results are shown in Table 6 and include a comparison to the model without using previous flare activity.

Space Radiation Analysis Group (SRAG) provided a list of >100 MeV, >1 pfu events and >50 MeV, >0.5 pfu events for verification purposes. Filament eruption events were removed from the event list. Also eliminated from the event list were events with >10 MeV, >10 pfu that
<caption>Table 6</caption>

Ratios and skill scores for MAG4 with and without flaring. PC = Percent Correct; POD = Probability of Detection; FAR = False Alarm Rate; HSS = Heidke Skill Score; TSS = True Skill Score.

<table><thead><tr><th>Mode</th><th>PC</th><th>POD</th><th>FAR</th><th>HSS</th><th>TSS</th></tr></thead><tbody><tr><td>MAG4 without flaring</td><td>95.5 ± 0.5</td><td>0.31 ± 0.06</td><td>0.50 ± 0.08</td><td>0.35 ± 0.05</td><td>0.47 ± 0.08</td></tr><tr><td>MAG4 with flaring</td><td>95.7 ± 0.5</td><td>0.38 ± 0.05</td><td>0.48 ± 0.06</td><td>0.42 ± 0.04</td><td>0.49 ± 0.06</td></tr></tbody></table>

had no enhancement at higher energy fluxes. These latter events were deemed to be operationally irrelevant.

### 3.14. MagPy - Magnetogram Forecast in Python

**Model developers:** David Falconer (University of Alabama at Huntsville) and Tilaye Tadesse, Yash Kadadi, and Ian Fernandes (developed at NASA/Johnson Space Center (JSC)/ SRAG).

**Model description:** MagPy is the Python version of MAG4. It is a near real-time large-database program for forecasting an AR's next-day production rate of major flares, CMEs, and SEP events from its free-energy proxy and short-term previous flare productivity. MAG4 is written in Interactive Data Language (IDL) and despite its success, the IDL platform upon which it was built may not serve robust development requirements that are necessary for the model's growth, suggesting a more modern and widespread programming language. With the explosion in popularity of Python, the number of packages and abundant online documentation has grown dramatically, resulting in functions that duplicate and greatly exceed much of IDL's capabilities.

MagPy is designed for the empirical forecasting of M- and X-class flares, CMEs, and SEP events. For each type of event, the algorithm is based on the empirical relationship between the event rate and the free magnetic energy proxy of each AR on the solar disk. The program uses vector magnetograms to measure these free energy proxies, including the gradient of the magnetic field across the neutral line, the magnetic shear angle across the neutral line, and others. These free energy proxies are then related to historically derived event rates, and with Poisson statistics, probabilities of the occurrence of M-class flares, X-class flares, CMEs, Fast CMEs, and SPEs within the next 24 h of a magnetogram are determined.

The proxy of free magnetic energy takes advantage of the observation that non-potential or sheared magnetic fields tend to build up along magnetic neutral lines, as well as the observation that magnetic neutral lines that have strong gradients across them in the vertical magnetic field nearly always have a strongly sheared horizontal field along them. In order to evaluate these proxies, we have to identify the strong field neutral line in the magnetic island. The neutral line (or polarity inversion line) in an AR's magnetogram is the line that separates positive and negative polarity flux, i.e., the zero-Gauss contour. We divide the zero-Gauss contour into segments, each roughly a pixel in length. For each segment, its length, neighbor

segments, the interpolated values of the potential horizontal field at the midpoint of the segment, the vertical gradient of the vertical field at the midpoint, the average positive field nearby, and the average negative field nearby are compiled. A segment is considered if it meets the following two conditions: (1) it has a strong potential horizontal field (greater than optimal potential magnetic field value) and (2) it separates at least moderately strong positive and negative fields (optimally-pixel-smoothed fluxes of both polarities are stronger than optimal field value). The potential horizontal field is calculated from the observed vertical component of an AR's magnetogram. The rules for average positive and negative field strength ensure that the segment separates active region polarities rather than an active region polarity and a quiet Sun polarity. The remaining segments are further checked for whether they have a neighbor segment; isolated segments are discarded. Overall MagPy data processing techniques are shown in Fig. 4.

In order to determine the optimal function relating event rate and total non-potentiality as well as the optimal data selection criteria and magnetic field thresholds to use in the forecasting process, data mining must be performed. The idea behind data mining is that data from many years can be processed and analyzed for their free magnetic energy proxies and other related parameters to determine the most accurate function relating an active region's total non-potentiality to its major solar flare event rate. We run the data mining process with thousands of magnetic field threshold configurations (see optimal thresholds previously mentioned) and compare skill scores across the validation results to determine the best set of magnetic field thresholds for operational use. In doing so, we have succeeded in improving the HSS skill score of 0.32 in MAG4 to 0.48 in MagPy.

For each SHARP, MagPy performs the following steps:

* Use an analytic solution to calculate the transverse potential field from the $B_z$ magnetic field at every pixel.

* Use a contour scheme to identify the neutral line. Apply a cut to the neutral line such that the transverse potential field must be greater than its optimal field value threshold and the averaged positive and negative field magnitudes must be greater than their optimal threshold to capture the neutral line between strong magnetic fields.

* Calculate the following magnetic parameters:

- Length of the strong-field neutral line.

- Area with $|B_z| > \text{optimal magnetic field value}$.
<caption>Table 6</caption>

Ratios and skill scores for MAG4 with and without flaring. PC = Percent Correct; POD = Probability of Detection; FAR = False Alarm Rate; HSS = Heidke Skill Score; TSS = True Skill Score.

<table><thead><tr><th>Mode</th><th>PC</th><th>POD</th><th>FAR</th><th>HSS</th><th>TSS</th></tr></thead><tbody><tr><td>MAG4 without flaring</td><td>95.5 ± 0.5</td><td>0.31 ± 0.06</td><td>0.50 ± 0.08</td><td>0.35 ± 0.05</td><td>0.47 ± 0.08</td></tr><tr><td>MAG4 with flaring</td><td>95.7 ± 0.5</td><td>0.38 ± 0.05</td><td>0.48 ± 0.06</td><td>0.42 ± 0.04</td><td>0.49 ± 0.06</td></tr></tbody></table>

had no enhancement at higher energy fluxes. These latter events were deemed to be operationally irrelevant.

### 3.14. MagPy - Magnetogram Forecast in Python

**Model developers:** David Falconer (University of Alabama at Huntsville) and Tilaye Tadesse, Yash Kadadi, and Ian Fernandes (developed at NASA/Johnson Space Center (JSC)/ SRAG).

**Model description:** MagPy is the Python version of MAG4. It is a near real-time large-database program for forecasting an AR's next-day production rate of major flares, CMEs, and SEP events from its free-energy proxy and short-term previous flare productivity. MAG4 is written in Interactive Data Language (IDL) and despite its success, the IDL platform upon which it was built may not serve robust development requirements that are necessary for the model's growth, suggesting a more modern and widespread programming language. With the explosion in popularity of Python, the number of packages and abundant online documentation has grown dramatically, resulting in functions that duplicate and greatly exceed much of IDL's capabilities.

MagPy is designed for the empirical forecasting of M- and X-class flares, CMEs, and SEP events. For each type of event, the algorithm is based on the empirical relationship between the event rate and the free magnetic energy proxy of each AR on the solar disk. The program uses vector magnetograms to measure these free energy proxies, including the gradient of the magnetic field across the neutral line, the magnetic shear angle across the neutral line, and others. These free energy proxies are then related to historically derived event rates, and with Poisson statistics, probabilities of the occurrence of M-class flares, X-class flares, CMEs, Fast CMEs, and SPEs within the next 24 h of a magnetogram are determined.

The proxy of free magnetic energy takes advantage of the observation that non-potential or sheared magnetic fields tend to build up along magnetic neutral lines, as well as the observation that magnetic neutral lines that have strong gradients across them in the vertical magnetic field nearly always have a strongly sheared horizontal field along them. In order to evaluate these proxies, we have to identify the strong field neutral line in the magnetic island. The neutral line (or polarity inversion line) in an AR's magnetogram is the line that separates positive and negative polarity flux, i.e., the zero-Gauss contour. We divide the zero-Gauss contour into segments, each roughly a pixel in length. For each segment, its length, neighbor

segments, the interpolated values of the potential horizontal field at the midpoint of the segment, the vertical gradient of the vertical field at the midpoint, the average positive field nearby, and the average negative field nearby are compiled. A segment is considered if it meets the following two conditions: (1) it has a strong potential horizontal field (greater than optimal potential magnetic field value) and (2) it separates at least moderately strong positive and negative fields (optimally-pixel-smoothed fluxes of both polarities are stronger than optimal field value). The potential horizontal field is calculated from the observed vertical component of an AR's magnetogram. The rules for average positive and negative field strength ensure that the segment separates active region polarities rather than an active region polarity and a quiet Sun polarity. The remaining segments are further checked for whether they have a neighbor segment; isolated segments are discarded. Overall MagPy data processing techniques are shown in Fig. 4.

In order to determine the optimal function relating event rate and total non-potentiality as well as the optimal data selection criteria and magnetic field thresholds to use in the forecasting process, data mining must be performed. The idea behind data mining is that data from many years can be processed and analyzed for their free magnetic energy proxies and other related parameters to determine the most accurate function relating an active region's total non-potentiality to its major solar flare event rate. We run the data mining process with thousands of magnetic field threshold configurations (see optimal thresholds previously mentioned) and compare skill scores across the validation results to determine the best set of magnetic field thresholds for operational use. In doing that, we have succeeded in improving the HSS skill score of 0.32 in MAG4 to 0.48 in MagPy.

For each SHARP, MagPy performs the following steps:

* Use an analytic solution to calculate the transverse potential field from the $B_z$ magnetic field at every pixel.

* Use a contour scheme to identify the neutral line. Apply a cut to the neutral line such that the transverse potential field must be greater than its optimal field value threshold and the averaged positive and negative field magnitudes must be greater than their optimal threshold to capture the neutral line between strong magnetic fields.

* Calculate the following magnetic parameters:

- Length of the strong-field neutral line.

- Area with $|B_z| > \text{optimal magnetic field value}$.
<figure><img src="image_5.png" /><figcaption>Fig. 4. MagPy Image Processing Pipeline: a succinct summary of the image processing techniques used to extract free-energy proxies.</figcaption></figure>

- Magnetic flux with $|B_z| > \text{optimal magnetic field value}$.

- Shear angle between the magnetic field and transverse potential field across each pixel on the neutral line.

- Gradient of the magnetic field across each pixel on the neutral line.

* Use the shear angle and gradient along the neutral line to get the event rate per day based on previously derived forecast curves for each event type.

* Use Poisson statistics to convert the event rate per day to a probability. This is also performed for the aggregate of event rates across all ARs currently on the disk to get a full-disk probability.

**Inputs:**

* Vector magnetograms from the SDO/HMI

* NOAA Solar Region Summary (SRS)

* CCMC-Database Of Notifications, Knowledge, Information (DONKI) Flare database

**Outputs:** Probabilities of occurrence of the following:

* M + X-class flares

* X-class flares

* CMEs

* Fast CMEs

* SPEs

**Free parameters:** None.

**Limitations and caveats:** Each magnetogram is also subject to multiple quality control checks put in place to increase the fidelity of the data analyzed. MagPy utilizes a 45-degree cone filter to discard magnetograms on the limbs of the solar disk. These magnetograms, distorted by effects like foreshortening, engender high degrees of error in free-energy proxy calculations and spawn false neutral-lines. As these types of magnetograms can severely skew the correlation between proxies and event-rates, they

are left out of the training dataset. MagPy is most accurate within a 45 degree cone from disk center.

**Access to model output or forecasts:** The model will be available at CCMC for Runs on Request, as well as on the SEP Probability Scoreboard.

**Model validation:** MagPy is actively under development, including the addition of SEP forecasting capabilities, and validation will be carried out after the model is finalized.

### 3.15. MEMPSEP - Multivariate Ensemble of Models for Probabilistic SEP prediction

**Model developers:** Maher A Dayeh (Southwest Research Institute (SWRI), University of Texas San Antonio), Andres Munoz-Jaramillo (SWRI Boulder), Hazel M. Bain (Cooperative Institute for Research in Environmental Sciences (CIRES)), Subhamoy Chatterjee (SWRI Boulder), Kimberly D. Moreland (University of Texas San Antonio, SWRI).

**Model description:** The Multivariate Ensemble of Models for Probabilistic SEP prediction (MEMPSEP) is based on a data-driven Convolutional Neural Network (CNN) (LeCun and Bengio, 1998) architecture that takes both remote images (SOHO/MDI and SDO/HMI full-disk magnetogram sequences for now) and in situ pre-flare to post-SEP parameters as inputs and provides (i) SEP occurrence probability and in the case this probability passes a certain threshold, it then provides (ii) absolute prediction of physical SEP properties (e.g., SEP peak flux), as illustrated in a flowchart (see Fig. 5).

This must be emphasized that the objective here is not to do just a binary classification. Instead, the focus is on estimating true probabilities of SEP occurrence with associated uncertainties (using a model-ensemble). The aim is to provide flexibility for the users of MEMPSEP forecast to determine their own acceptable level of risk, rather than imposing a threshold of detection that optimizes an arbitrary binary classification metric. Furthermore, instead of training a single model, the class imbalance between negative and positive events is utilized to train an ensemble of
<figure><img src="image_5.png" /><figcaption>Fig. 4. MagPy Image Processing Pipeline: a succinct summary of the image processing techniques used to extract free-energy proxies.</figcaption></figure>

- Magnetic flux with $|B_z| > \text{optimal magnetic field value}$.

- Shear angle between the magnetic field and transverse potential field across each pixel on the neutral line.

- Gradient of the magnetic field across each pixel on the neutral line.

* Use the shear angle and gradient along the neutral line to get the event rate per day based on previously derived forecast curves for each event type.

* Use Poisson statistics to convert the event rate per day to a probability. This is also performed for the aggregate of event rates across all ARs currently on the disk to get a full-disk probability.

**Inputs:**

* Vector magnetograms from the SDO/HMI

* NOAA Solar Region Summary (SRS)

* CCMC-Database Of Notifications, Knowledge, Information (DONKI) Flare database

**Outputs:** Probabilities of occurrence of the following:

* M + X-class flares

* X-class flares

* CMEs

* Fast CMEs

* SPEs

**Free parameters:** None.

**Limitations and caveats:** Each magnetogram is also subject to multiple quality control checks put in place to increase the fidelity of the data analyzed. MagPy utilizes a 45-degree cone filter to discard magnetograms on the limbs of the solar disk. These magnetograms, distorted by effects like foreshortening, engender high degrees of error in free-energy proxy calculations and spawn false neutral-lines. As these types of magnetograms can severely skew the correlation between proxies and event-rates, they

are left out of the training dataset. MagPy is most accurate within a 45 degree cone from disk center.

**Access to model output or forecasts:** The model will be available at CCMC for Runs on Request, as well as on the SEP Probability Scoreboard.

**Model validation:** MagPy is actively under development, including the addition of SEP forecasting capabilities, and validation will be carried out after the model is finalized.

### 3.15. MEMPSEP - Multivariate Ensemble of Models for Probabilistic SEP prediction

**Model developers:** Maher A Dayeh (Southwest Research Institute (SWRI), University of Texas San Antonio), Andres Munoz-Jaramillo (SWRI Boulder), Hazel M. Bain (Cooperative Institute for Research in Environmental Sciences (CIRES)), Subhamoy Chatterjee (SWRI Boulder), Kimberly D. Moreland (University of Texas San Antonio, SWRI).

**Model description:** The Multivariate Ensemble of Models for Probabilistic SEP prediction (MEMPSEP) is based on a data-driven Convolutional Neural Network (CNN) (LeCun and Bengio, 1998) architecture that takes both remote images (SOHO/MDI and SDO/HMI full-disk magnetogram sequences for now) and in situ pre-flare to post-SEP parameters as inputs and provides (i) SEP occurrence probability and in the case this probability passes a certain threshold, it then provides (ii) absolute prediction of physical SEP properties (e.g., SEP peak flux), as illustrated in a flowchart (see Fig. 5).

This must be emphasized that the objective here is not to do just a binary classification. Instead, the focus is on estimating true probabilities of SEP occurrence with associated uncertainties (using a model-ensemble). The aim is to provide flexibility for the users of MEMPSEP forecast to determine their own acceptable level of risk, rather than imposing a threshold of detection that optimizes an arbitrary binary classification metric. Furthermore, instead of training a single model, the class imbalance between negative and positive events is utilized to train an ensemble of
<figure><img src="image_6.png" /><figcaption>Fig. 5. Architecture of MEMPSEP model.</figcaption></figure>

models. This has the added benefit of providing a clear measure of uncertainty in the forecast. The predicted SEP occurrence probability of each ensemble model is calibrated (Guo et al., 2017) such that the probabilistic forecast is reliable (Bain et al., 2021) to use.

The design of the test set is tailored to be non-modulated by the solar cycle and the model inferences are evaluated with uncertainty for each event within the test set. Based on the median of the ensemble and a probability threshold of 0.5, a few classification metrics (Bain et al., 2021) are estimated, reported at the end of this section.

For SEP peak prediction task, a similar CNN architecture, as in prediction of SEP occurrence probability, is trained. The layers are frozen for probability estimation branch and those probabilities are used to penalize the loss function for training the SEP peak prediction layers (Fig. 5). This coupling is done to ensure tighter prediction of SEP peak for the events with higher predicted probabilities.

The dataset driving MEMPSEP is a multivariate remote and in situ dataset. The main event list is based on the GOES flare list from the Heliospheric Event Knowledgebase (HEK) and contains over 17,000 flare events. For each flare event, remote solar images along with spacecraft measured X-ray, radio, proton, electron, upstream Interplanetary (IP) plasma and magnetic field properties are included, in addition to associated IP coronal mass ejections and IP shock parameters, whenever available. The in situ data comes from multiple instruments onboard different operational missions, including GOES, Advanced Composition Explorer

(ACE), and Wind spacecraft. The remote sensing data, also from various instruments, comprises full-disk magnetograms, EUV, and coronagraph images from STEREO, SOHO, and SDO. Positive SEP events are selected based on a defined threshold using the integrated proton fluxes. Physical SEP properties such as SEP duration, peak flux, elemental composition, and fluence for each event are also derived and included. Averaged elemental composition of the suprathermal particles (Dayeh et al., 2009; Dayeh et al., 2017) prior to each event by up to 3 days is also calculated and included as an input. Special consideration is given to data that is currently obtained in operational real-time or that will be available in real-time on upcoming missions.

**Inputs:** Current form of MEMPSEP ingests the following inputs-

* Full-disc (SOHO/MDI + SDO/HMI) magnetogram sequences spanning over 3 days prior to flare-onset with a temporal resolution of 6 h.

* Pre-flare upstream properties such as solar wind plasma (speed, density), interplanetary magnetic field components and elemental abundances etc.

* GOES X-ray pre-flare time series for 24 h with a temporal resolution of 1 min.

* GOES Electron pre-flare time series for 24 h with a temporal resolution of 5 min.

The model has capability to also seamlessly ingest inputs such as SOHO/Extreme ultraviolet Imaging Telescope (EIT) 171Å + SDO/AIA 171Å image sequences, SOHO/
<figure><img src="image_6.png" /><figcaption>Fig. 5. Architecture of MEMPSEP model.</figcaption></figure>

models. This has the added benefit of providing a clear measure of uncertainty in the forecast. The predicted SEP occurrence probability of each ensemble model is calibrated (Guo et al., 2017) such that the probabilistic forecast is reliable (Bain et al., 2021) to use.

The design of the test set is tailored to be non-modulated by the solar cycle and the model inferences are evaluated with uncertainty for each event within the test set. Based on the median of the ensemble and a probability threshold of 0.5, a few classification metrics (Bain et al., 2021) are estimated, reported at the end of this section.

For SEP peak prediction task, a similar CNN architecture, as in prediction of SEP occurrence probability, is trained. The layers are frozen for probability estimation branch and those probabilities are used to penalize the loss function for training the SEP peak prediction layers (Fig. 5). This coupling is done to ensure tighter prediction of SEP peak for the events with higher predicted probabilities.

The dataset driving MEMPSEP is a multivariate remote and in situ dataset. The main event list is based on the GOES flare list from the Heliospheric Event Knowledgebase (HEK) and contains over 17,000 flare events. For each flare event, remote solar images along with spacecraft measured X-ray, radio, proton, electron, upstream Interplanetary (IP) plasma and magnetic field properties are included, in addition to associated IP coronal mass ejections and IP shock parameters, whenever available. The in situ data comes from multiple instruments onboard different operational missions, including GOES, Advanced Composition Explorer

(ACE), and Wind spacecraft. The remote sensing data, also from various instruments, comprises full-disk magnetograms, EUV, and coronagraph images from STEREO, SOHO, and SDO. Positive SEP events are selected based on a defined threshold using the integrated proton fluxes. Physical SEP properties such as SEP duration, peak flux, elemental composition, and fluence for each event are also derived and included. Averaged elemental composition of the suprathermal particles (Dayeh et al., 2009; Dayeh et al., 2017) prior to each event by up to 3 days is also calculated and included as an input. Special consideration is given to data that is currently obtained in operational real-time or that will be available in real-time on upcoming missions.

**Inputs:** Current form of MEMPSEP ingests the following inputs-

* Full-disc (SOHO/MDI + SDO/HMI) magnetogram sequences spanning over 3 days prior to flare-onset with a temporal resolution of 6 h.

* Pre-flare upstream properties such as solar wind plasma (speed, density), interplanetary magnetic field components and elemental abundances etc.

* GOES X-ray pre-flare time series for 24 h with a temporal resolution of 1 min.

* GOES Electron pre-flare time series for 24 h with a temporal resolution of 5 min.

The model has capability to also seamlessly ingest inputs such as SOHO/Extreme ultraviolet Imaging Telescope (EIT) 171Å + SDO/AIA 171Å image sequences, SOHO/
LASCO-C2 + C3 image sequences and WIND/WAVES data reflecting radio bursts.

**Outputs:** MEMPSEP performs both classification and regression tasks. For every set of (remote + in situ) inputs MEMPSEP produces following outcomes:

* SEP occurrence probability and associated uncertainty

* SEP properties (e.g., peak flux, fluence, event spectrum, time to peak, duration etc.) and associated uncertainty for user defined energy range

**Free parameters:** The prediction model-ensemble does not have tunable parameters that need to be fine-tuned by the user/forecaster. However, it relies on the training of an ensemble of neural networks which involves the creation of training, validation, and test sets; definition of a neural network architecture, and choosing a neural network optimizer with hyperparameters.

**Limitations and caveats:** CNNs are data hungry algorithms, so they need to be trained and deployed very carefully to avoid overfitting. This is marked as the limitation/caveat because there is a series of good practices that need to be followed strictly in order for the results to be meaningful. This includes:

1. a strict separation between training + validation and test sets.

2. a disciplined optimization of the experiments exclusively on training + validation sets.

3. to never make architecture or training decisions based on the final results of deploying on the test set.

Furthermore, MEMPSEP currently comprises of an ensemble of 10 models. However, it is unknown what is the critical number of ensemble members after which the uncertainty in predicted SEP occurrence probability and peak flux converges. MEMPSEP group is currently working on that aspect and also investigating the effect of additional inputs on the prediction.

**Access to model output or forecasts:** The model is currently not available to the public.

**Model validation:** The validation of MEMPSEP is done on a test set comprising of unseen events tied to different flare classes (C, M, X) and spanning over the years 1998–2013. Based on the median of the model-ensemble prediction and a probability threshold of 0.5 (as used in different SEP classification models, Bain et al., 2021), a few classification metrics are evaluated. The POD (TP/TP + FN) and False Positive Rate (FP/FP + TN) are found to be 0.79 and 0.27 respectively, TSS and HSS are to be 0.52 and 0.43 respectively. The metrics, namely AUC (ROC) and Brier Score (BS), that do not depend on a single classification threshold, are also estimated. AUC and BS are found to be 0.85 and 0.14 respectively. The regression step provides a statistically significant correlation of ~0.55 between target and predicted median peaks for positively predicted events.

### 3.16. M-FLAMPA - Multiple-Field-Line-Advection Model for Particle Acceleration

**Model developers:** Igor V. Sokolov, Lulu Zhao, Zhen-guang Huang, Dmitry Borovikov, Ilia Roussev, Aleksandre Taktashvili, Frederic Effenberger, and Tamas Gombosi (developed at the University of Michigan).

**Model description:** The SEP forecast model, Multiple Field Line Advection Model for Particle Acceleration (M-FLAMPA), is composed of several fully coupled modules in the SWMF developed at the University of Michigan (Borovikov et al., 2018). There are three components in this SEP forecast model: 1) background solar wind, 2) coronal mass ejections, and 3) particle acceleration and transport. The background solar wind plasma in which the SEPs propagate is modeled by the AWSOM-Realtime (AWSOM(-R)) driven by the near-real-time hourly updated GONG (bihourly Air Force Data Assimilation Photospheric flux Transport (ADAPT)-GONG) magnetogram. In the background solar wind, real/hypothetical CMEs are regularly launched employing the Eruptive Event Generator using Gibson-Low configuration (EEGGL), by inserting a flux rope estimated from the free magnetic energy in the active region. The acceleration and transport processes are then modeled self-consistently by the multiple magnetic field line tracker (MFLAMPA). The three modules are now fully integrated and run faster than real-time, thus providing a forecast capability.

The 3D global solar wind plasma in the Solar Corona (1–20*R<sub>s</sub>), Inner Heliosphere (20*R<sub>s</sub> – 2 AU) and Middle Heliosphere (2 AU – 20 AU) are modeled by the AWSOM(-R) configuration in SWMF (Sokolov et al., 2013; Sokolov et al., 2021; Gombosi et al., 2018; Gombosi et al., 2021). AWSOM(-R) is an Aflvén wave-driven, self-consistent solar wind model, in which the coronal plasma is heated by the dissipation of two discrete turbulence populations propagating parallel and antiparallel to the magnetic field (Sokolov et al., 2013). AWSOM applies a semi-empirical coronal heating function that is based on these aspects of the physics of Alfvén waves. The fully 3D version of AWSOM is computationally expensive. And AWSOM-R is a version that runs faster than real-time by reducing the 3D problem to a 1D problem by bringing boundary conditions up from the solar surface along 1D threads and adopting the simplifying assumption that the magnetic field may be considered to be potential within a certain radial distance from the Sun.

The CME/Coronal Mass Ejection (ICME) simulation in this SEP forecast model is driven by the EEGGL module in SWMF (Jin et al., 2017; Borovikov et al., 2017). The initial conditions of the CME within the solar corona is treated by inserting an unstable (or force imbalanced) flux rope suggested by Gibson and Low (1998) into an active region of the solar corona. The magnetogram from GONG and the observed CME speed (from CDAW catalog and/or DONKI database) are used to calculate the flux rope parameters. This approach offers a relatively simple, and
LASCO-C2 + C3 image sequences and WIND/WAVES data reflecting radio bursts.

**Outputs:** MEMPSEP performs both classification and regression tasks. For every set of (remote + in situ) inputs MEMPSEP produces following outcomes:

* SEP occurrence probability and associated uncertainty

* SEP properties (e.g., peak flux, fluence, event spectrum, time to peak, duration etc.) and associated uncertainty for user defined energy range

**Free parameters:** The prediction model-ensemble does not have tunable parameters that need to be fine-tuned by the user/forecaster. However, it relies on the training of an ensemble of neural networks which involves the creation of training, validation, and test sets; definition of a neural network architecture, and choosing a neural network optimizer with hyperparameters.

**Limitations and caveats:** CNNs are data hungry algorithms, so they need to be trained and deployed very carefully to avoid overfitting. This is marked as the limitation/caveat because there is a series of good practices that need to be followed strictly in order for the results to be meaningful. This includes:

1. a strict separation between training + validation and test sets.

2. a disciplined optimization of the experiments exclusively on training + validation sets.

3. to never make architecture or training decisions based on the final results of deploying on the test set.

Furthermore, MEMPSEP currently comprises of an ensemble of 10 models. However, it is unknown what is the critical number of ensemble members after which the uncertainty in predicted SEP occurrence probability and peak flux converges. MEMPSEP group is currently working on that aspect and also investigating the effect of additional inputs on the prediction.

**Access to model output or forecasts:** The model is currently not available to the public.

**Model validation:** The validation of MEMPSEP is done on a test set comprising of unseen events tied to different flare classes (C, M, X) and spanning over the years 1998–2013. Based on the median of the model-ensemble prediction and a probability threshold of 0.5 (as used in different SEP classification models, Bain et al., 2021), a few classification metrics are evaluated. The POD (TP/TP + FN) and False Positive Rate (FP/FP + TN) are found to be 0.79 and 0.27 respectively, TSS and HSS are to be 0.52 and 0.43 respectively. The metrics, namely AUC (ROC) and Brier Score (BS), that do not depend on a single classification threshold, are also estimated. AUC and BS are found to be 0.85 and 0.14 respectively. The regression step provides a statistically significant correlation of ~0.55 between target and predicted median peaks for positively predicted events.

### 3.16. M-FLAMPA - Multiple-Field-Line-Advection Model for Particle Acceleration

**Model developers:** Igor V. Sokolov, Lulu Zhao, Zhen-guang Huang, Dmitry Borovikov, Ilia Roussev, Aleksandre Taktashvili, Frederic Effenberger, and Tamas Gombosi (developed at the University of Michigan).

**Model description:** The SEP forecast model, Multiple Field Line Advection Model for Particle Acceleration (M-FLAMPA), is composed of several fully coupled modules in the SWMF developed at the University of Michigan (Borovikov et al., 2018). There are three components in this SEP forecast model: 1) background solar wind, 2) coronal mass ejections, and 3) particle acceleration and transport. The background solar wind plasma in which the SEPs propagate is modeled by the AWSOM-Realtime (AWSOM(-R)) driven by the near-real-time hourly updated GONG (bihourly Air Force Data Assimilation Photospheric flux Transport (ADAPT)-GONG) magnetogram. In the background solar wind, real/hypothetical CMEs are regularly launched employing the Eruptive Event Generator using Gibson-Low configuration (EEGGL), by inserting a flux rope estimated from the free magnetic energy in the active region. The acceleration and transport processes are then modeled self-consistently by the multiple magnetic field line tracker (MFLAMPA). The three modules are now fully integrated and run faster than real-time, thus providing a forecast capability.

The 3D global solar wind plasma in the Solar Corona (1–20*R<sub>s</sub>), Inner Heliosphere (20*R<sub>s</sub> – 2 AU) and Middle Heliosphere (2 AU – 20 AU) are modeled by the AWSOM(-R) configuration in SWMF (Sokolov et al., 2013; Sokolov et al., 2021; Gombosi et al., 2018; Gombosi et al., 2021). AWSOM(-R) is an Aflvén wave-driven, self-consistent solar wind model, in which the coronal plasma is heated by the dissipation of two discrete turbulence populations propagating parallel and antiparallel to the magnetic field (Sokolov et al., 2013). AWSOM applies a semi-empirical coronal heating function that is based on these aspects of the physics of Alfvén waves. The fully 3D version of AWSOM is computationally expensive. And AWSOM-R is a version that runs faster than real-time by reducing the 3D problem to a 1D problem by bringing boundary conditions up from the solar surface along 1D threads and adopting the simplifying assumption that the magnetic field may be considered to be potential within a certain radial distance from the Sun.

The CME/Coronal Mass Ejection (ICME) simulation in this SEP forecast model is driven by the EEGGL module in SWMF (Jin et al., 2017; Borovikov et al., 2017). The initial conditions of the CME within the solar corona is treated by inserting an unstable (or force imbalanced) flux rope suggested by Gibson and Low (1998) into an active region of the solar corona. The magnetogram from GONG and the observed CME speed (from CDAW catalog and/or DONKI database) are used to calculate the flux rope parameters. This approach offers a relatively simple, and
inexpensive model for CME initiation based on empirical features of pre-event conditions (Gombosi et al., 2021). The EEGGL module is publicly available for download at http://csem.engin.umich.edu or can be used through CCMC (https://ccmc.gsfc.nasa.gov/eeggl/).

In the proposed SEP forecast model, protons are accelerated at the shocks driven by the CMEs through first order Fermi acceleration mechanism (e.g., Axford et al., 1977; Krymsky, 1977). The original FLAMPA scheme was developed by Sokolov et al. (2004). In this early version the energetic particle distribution function was solved along a single time-evolving magnetic field line. MFLAMPA is a high-performance extension of FLAMPA, enabling us to solve the kinetic equation of SEPs along a multitude of magnetic field lines simultaneously. The line-extracting procedure is augmented with an algorithm designed to interpolate the fields sampled on block adaptive grids (Borovikov et al., 2015) that eliminates spurious distortions near grid resolution interfaces that routinely occur in large-scale MHD grids. Such artifacts could result in unphysical particle acceleration, but they are thus avoided in the model. In addition, novel mathematical methods have been applied to the extracted magnetic field lines to sharpen the shocks thus making the Fermi acceleration process to be more efficient.

MFLAMPA is directly coupled with AWSOM-R and EEGGL within the SWMF. Magnetic field lines are extracted seamlessly from the active region in the solar corona through the inner and middle heliosphere to the observer and are passed to MFLAMPA. MFLAMPA then calculates the evolving energetic particle populations by solving the kinetic equations. MFLAMPA incorporates the effects of the near-Sun field line meandering that affects the perpendicular transport of SEPs and can explain the occurrence of large longitudinal spread observed even in the early phases of such events.

**Inputs:** The Input to the model includes the magnetogram that drives the background solar wind simulations, e.g., GONG, HMI, and ADAPT-GONG. The CME speed, together with the magnetogram is needed to calculate the parameters of the flux rope that will be inserted into the corresponding active region.

**Outputs:** Since the MFLAMPA model calculates the distribution function of energetic particles within the energy range of 10 keV to 1 GeV, the model is able to output the time intensity profiles, at energies >10 MeV, >50 MeV, >100 MeV, or otherwise specified by the user, along each individual magnetic field line extracted using MFLAMPA. Moreover, an ensemble average and standard deviation of the intensity profiles on the magnetic field lines around the observational point of interest is also provided. The intensity at the prescribed energies on the longitude and latitude sphere at certain radial distance is also provided.

**Free parameters:** The free parameters of the model consist of the treatment of particles' injection, i.e., seed populations into the shock system. In the current model, we

follow the seed particle setup in Sokolov et al. (2004), assuming a suprathermal tail of protons from the thermal distribution at the local kinetic temperature through the injection energy, which is a free parameter. The injection energy is set to be 10 keV. We are currently working on scaling the suprathermal seed particle distribution observed at 1 AU<fnref n="11" /> back to the sun and extract the seed population.

**Limitations and caveats:** The integrated AWSOM(-R), EEGGL and MFLAMPA model is time and computation resource consuming. When the CME propagates in the solar corona domain (SC domain, $1R_s \sim 20R_s$), the integrated model runs as fast as the real time using 4000 cpu cores. The most time consuming component is the MHD simulation of the CME propagation in the SC. When the CME left the SC domain, the integrated model runs much faster than real time. Another limitation of the model is the specification of the injected seed population.

**Access to model output or forecasts:** Users will be able to access MFLAMPA and associated models on CCMC's runs-on-request system (https://ccmc.gsfc.nasa.gov/requests/requests.php).

**Model validation:** The background solar wind model is validated by comparing the simulations and observations for both in situ solar wind and the predicted LOS appearance of the corona in different wavelengths. A detailed validation study of the solar wind model can be found in (Sachdeva et al., 2019; Gombosi et al., 2021). The EEGGL model to initialize CMEs and the subsequent CME/ICME evolution has been extensively used and validated (Jin et al., 2017, Manchester et al., 2004, 2005, 2008, 2012, 2014, Manchester et al., 2017, Roussev et al., 2004, Roussev, 2008, van der Holst et al., 2007, 2009). Borovikov et al. (2018) showed a proof-of-concept demonstration of AWSOM-R, EEGGL, and MFLAMPA for the January 23, 2012 SEP event. The simulated >10 MeV time profile is plotted alongside GOES, however, no further analysis is performed.

## 3.17. PARADISE - PArticle Radiation Asset Directed at Interplanetary Space Exploration

**Model developers:** Nicolas Wijsen (Katholieke Universiteit (KU) Leuven), Angels Aran (Universitat de Barcelona), Stefaan Poedts (KU Leuven, University of Maria Curie-Skłodowska).

**Model description:** PArticle Radiation Asset Directed at Interplanetary Space Exploration (PARADISE; Wijsen, 2020) is a physics-based model aimed at simulating the acceleration and transport of energetic particles in the inner heliosphere. This is done by evolving energetic particle distributions through a solar wind configuration generated by the time-dependent three-dimensional MHD model named European Heliospheric FORecasting Information

<footnote n="11">¹¹ https://ccmc.gsfc.nasa.gov/assessment/topics/SEP/campaign2020/seed population.php.</footnote>
inexpensive model for CME initiation based on empirical features of pre-event conditions (Gombosi et al., 2021). The EEGGL module is publicly available for download at http://csem.engin.umich.edu or can be used through CCMC (https://ccmc.gsfc.nasa.gov/eeggl/).

In the proposed SEP forecast model, protons are accelerated at the shocks driven by the CMEs through first order Fermi acceleration mechanism (e.g., Axford et al., 1977; Krymsky, 1977). The original FLAMPA scheme was developed by Sokolov et al. (2004). In this early version the energetic particle distribution function was solved along a single time-evolving magnetic field line. MFLAMPA is a high-performance extension of FLAMPA, enabling us to solve the kinetic equation of SEPs along a multitude of magnetic field lines simultaneously. The line-extracting procedure is augmented with an algorithm designed to interpolate the fields sampled on block adaptive grids (Borovikov et al., 2015) that eliminates spurious distortions near grid resolution interfaces that routinely occur in large-scale MHD grids. Such artifacts could result in unphysical particle acceleration, but they are thus avoided in the model. In addition, novel mathematical methods have been applied to the extracted magnetic field lines to sharpen the shocks thus making the Fermi acceleration process to be more efficient.

MFLAMPA is directly coupled with AWSOM-R and EEGGL within the SWMF. Magnetic field lines are extracted seamlessly from the active region in the solar corona through the inner and middle heliosphere to the observer and are passed to MFLAMPA. MFLAMPA then calculates the evolving energetic particle populations by solving the kinetic equations. MFLAMPA incorporates the effects of the near-Sun field line meandering that affects the perpendicular transport of SEPs and can explain the occurrence of large longitudinal spread observed even in the early phases of such events.

**Inputs:** The Input to the model includes the magnetogram that drives the background solar wind simulations, e.g., GONG, HMI, and ADAPT-GONG. The CME speed, together with the magnetogram is needed to calculate the parameters of the flux rope that will be inserted into the corresponding active region.

**Outputs:** Since the MFLAMPA model calculates the distribution function of energetic particles within the energy range of 10 keV to 1 GeV, the model is able to output the time intensity profiles, at energies >10 MeV, >50 MeV, >100 MeV, or otherwise specified by the user, along each individual magnetic field line extracted using MFLAMPA. Moreover, an ensemble average and standard deviation of the intensity profiles on the magnetic field lines around the observational point of interest is also provided. The intensity at the prescribed energies on the longitude and latitude sphere at certain radial distance is also provided.

**Free parameters:** The free parameters of the model consist of the treatment of particles' injection, i.e., seed populations into the shock system. In the current model, we

follow the seed particle setup in Sokolov et al. (2004), assuming a suprathermal tail of protons from the thermal distribution at the local kinetic temperature through the injection energy, which is a free parameter. The injection energy is set to be 10 keV. We are currently working on scaling the suprathermal seed particle distribution observed at 1 AU<fnref n="11" /> back to the sun and extract the seed population.

**Limitations and caveats:** The integrated AWSOM(-R), EEGGL and MFLAMPA model is time and computation resource consuming. When the CME propagates in the solar corona domain (SC domain, $1R_s \sim 20R_s$), the integrated model runs as fast as the real time using 4000 cpu cores. The most time consuming component is the MHD simulation of the CME propagation in the SC. When the CME left the SC domain, the integrated model runs much faster than real time. Another limitation of the model is the specification of the injected seed population.

**Access to model output or forecasts:** Users will be able to access MFLAMPA and associated models on CCMC's runs-on-request system (https://ccmc.gsfc.nasa.gov/requests/requests.php).

**Model validation:** The background solar wind model is validated by comparing the simulations and observations for both in situ solar wind and the predicted LOS appearance of the corona in different wavelengths. A detailed validation study of the solar wind model can be found in (Sachdeva et al., 2019; Gombosi et al., 2021). The EEGGL model to initialize CMEs and the subsequent CME/ICME evolution has been extensively used and validated (Jin et al., 2017, Manchester et al., 2004, 2005, 2008, 2012, 2014, Manchester et al., 2017, Roussev et al., 2004, Roussev, 2008, van der Holst et al., 2007, 2009). Borovikov et al. (2018) showed a proof-of-concept demonstration of AWSOM-R, EEGGL, and MFLAMPA for the January 23, 2012 SEP event. The simulated >10 MeV time profile is plotted alongside GOES, however, no further analysis is performed.

### 3.17. PARADISE - PArticle Radiation Asset Directed at Interplanetary Space Exploration

**Model developers:** Nicolas Wijsen (Katholieke Universiteit (KU) Leuven), Angels Aran (Universitat de Barcelona), Stefaan Poedts (KU Leuven, University of Maria Curie-Skłodowska).

**Model description:** PArticle Radiation Asset Directed at Interplanetary Space Exploration (PARADISE; Wijsen, 2020) is a physics-based model aimed at simulating the acceleration and transport of energetic particles in the inner heliosphere. This is done by evolving energetic particle distributions through a solar wind configuration generated by the time-dependent three-dimensional MHD model named European Heliospheric FORecasting Information

<footnote n="11">¹¹ https://ccmc.gsfc.nasa.gov/assessment/topics/SEP/campaign2020/seedPopulation.php.</footnote>
Asset (EUHFORIA; Pomoell and Poedts, 2018). The coupling of PARADISE to EUHFORIA allows the model to simulate the effect of large-scale solar wind structures on the transport of energetic particles in the heliosphere. For example: [1] Wijsen et al. (2019b) and Wijsen et al. (2019a) studied how a high speed stream and a Co-rotating Interaction Region (CIR) can affect the transport and energization of impulsive SEP events. [2] Wijsen et al. (2022) simulated the particle transport and acceleration in an CME-driven shock wave, to reproduce the ESP event on 14 July 2012. An example PARADISE simulation for 9 October 2021 is shown in Fig. 6.

PARADISE is a Monte Carlo code that solves the five-dimensional Focused Transport Equation (FTE, e.g., le Roux and Webb, 2009) by integrating an equivalent set of stochastic differential equations forward in time using the Itô calculus (Gardiner, 2004). The FTE accounts for the particles' pitch-angle and momentum changes in compressional, shear, and accelerating solar wind flows, which are omni-present in EUHFORIA solar wind simulations. In addition, the FTE solved in PARADISE models the effect of small-scale turbulence on the energetic particle transport by including a spatial diffusion process perpendicular to the IMF and a pitch-angle diffusion process. Various different pitch-angle and perpendicular diffusion theories are available for use in PARADISE, which is made possible by its modular implementation structure (see Wijsen, 2020, for a full overview). The modular implementation of the code makes it also straightforward to add new diffusion theories to the model in the future. Moreover, the energetic particle's guiding center drifts due to magnetic gradients and the curvature of the IMF can be

included in the PARADISE simulations (e.g., Wijsen et al., 2020; van den Berg et al., 2021). Finally, we note that Chapter 3 of Wijsen (2020) describes the implementation of the PARADISE model in detail.

**Inputs:** PARADISE requires three different kinds of inputs:

1. EUHFORIA data files, which contain the solar wind velocity and magnetic field vectors on a spherical grid. As input, EUHFORIA simulations use synoptic magnetograms provided by the GONG. In addition, imaging observations can be used to constrain the input parameters of the CMEs that can be injected into the EUHFORIA simulation.

2. Particle injection distribution: To run PARADISE, the user needs to provide a particle injection distribution which can have both a spatial and a temporal dependence. Moreover, the injection distribution can depend on the properties of the background solar wind. For example, in Wijsen et al. (2021) and Wijsen et al. (2022), the particle injection distribution was assumed to be non-zero only at some of the interplanetary shock waves present in the simulation and, in addition, the injection distribution was scaled proportionally to the divergence of the solar wind velocity vector.

3. A configuration file which contains the information about the particle type (electrical charge and rest mass), the particle diffusion conditions, and the resolution of the output grid.

**Outputs:** PARADISE provides the directional particle intensities as a function of time on a five-dimensional grid.

<figure><img src="image_7.png" /><figcaption>Fig. 6. An example of a PARADISE simulation for the 9 October 2021 event. The left panel shows, in orange shades, a snapshot of the 1.1 – 1.9 MeV proton intensities. The background shows the radial speed profile of a EUHFORIA solar wind simulation, including a CME. The right panel shows the simulated time-intensity profiles (top panel), solar wind speed (middle panel), and solar wind magnetic field (bottom panel) at the location of Solar Orbiter.</figcaption></figure>
Asset (EUHFORIA; Pomoell and Poedts, 2018). The coupling of PARADISE to EUHFORIA allows the model to simulate the effect of large-scale solar wind structures on the transport of energetic particles in the heliosphere. For example: [1] Wijsen et al. (2019b) and Wijsen et al. (2019a) studied how a high speed stream and a Co-rotating Interaction Region (CIR) can affect the transport and energization of impulsive SEP events. [2] Wijsen et al. (2022) simulated the particle transport and acceleration in an CME-driven shock wave, to reproduce the ESP event on 14 July 2012. An example PARADISE simulation for 9 October 2021 is shown in Fig. 6.

PARADISE is a Monte Carlo code that solves the five-dimensional Focused Transport Equation (FTE, e.g., le Roux and Webb, 2009) by integrating an equivalent set of stochastic differential equations forward in time using the Itô calculus (Gardiner, 2004). The FTE accounts for the particles' pitch-angle and momentum changes in compressional, shear, and accelerating solar wind flows, which are omni-present in EUHFORIA solar wind simulations. In addition, the FTE solved in PARADISE models the effect of small-scale turbulence on the energetic particle transport by including a spatial diffusion process perpendicular to the IMF and a pitch-angle diffusion process. Various different pitch-angle and perpendicular diffusion theories are available for use in PARADISE, which is made possible by its modular implementation structure (see Wijsen, 2020, for a full overview). The modular implementation of the code makes it also straightforward to add new diffusion theories to the model in the future. Moreover, the energetic particle's guiding center drifts due to magnetic gradients and the curvature of the IMF can be

included in the PARADISE simulations (e.g., Wijsen et al., 2020; van den Berg et al., 2021). Finally, we note that Chapter 3 of Wijsen (2020) describes the implementation of the PARADISE model in detail.

**Inputs:** PARADISE requires three different kinds of inputs:

1. EUHFORIA data files, which contain the solar wind velocity and magnetic field vectors on a spherical grid. As input, EUHFORIA simulations use synoptic magnetograms provided by the GONG. In addition, imaging observations can be used to constrain the input parameters of the CMEs that can be injected into the EUHFORIA simulation.

2. Particle injection distribution: To run PARADISE, the user needs to provide a particle injection distribution which can have both a spatial and a temporal dependence. Moreover, the injection distribution can depend on the properties of the background solar wind. For example, in Wijsen et al. (2021) and Wijsen et al. (2022), the particle injection distribution was assumed to be non-zero only at some of the interplanetary shock waves present in the simulation and, in addition, the injection distribution was scaled proportionally to the divergence of the solar wind velocity vector.

3. A configuration file which contains the information about the particle type (electrical charge and rest mass), the particle diffusion conditions, and the resolution of the output grid.

**Outputs:** PARADISE provides the directional particle intensities as a function of time on a five-dimensional grid.

<figure><img src="image_7.png" /><figcaption>Fig. 6. An example of a PARADISE simulation for the 9 October 2021 event. The left panel shows, in orange shades, a snapshot of the 1.1 – 1.9 MeV proton intensities. The background shows the radial speed profile of a EUHFORIA solar wind simulation, including a CME. The right panel shows the simulated time-intensity profiles (top panel), solar wind speed (middle panel), and solar wind magnetic field (bottom panel) at the location of Solar Orbiter.</figcaption></figure>
The first three dimensions of the grid correspond to the
spatial dimensions and cover the inner heliosphere. The
two remaining dimensions of the grid correspond to the
energy and the pitch-angle coordinate of the energetic par-
ticles. The resolution of the output grid and the energy cov-
erage is defined by the user in the configuration file.

The model also provides the directional particle intensi-
ties, the first-order parallel anisotropies, and the omnidirec-
tional particle intensities at the location of the planets and
most spacecraft, including the STEREO spacecraft, PSP,
SO, and BepiColombo. In addition, the user can add addi-
tional virtual spacecraft to the simulations.

Finally, PARADISE contains python scripts and Jupy-
ter Notebooks which can be used to visualize the simula-
tion output.

**Free parameters:** PARADISE incorporates several dif-
ferent diffusion theories, which all have various free param-
eters. For example, the implemented pitch-angle diffusion
theories are derived from Quasi-Linear Theory (QLT;
Jokipii, 1966), and require the user to specify the properties
of the slab component of the solar wind magnetic turbu-
lence. As a default, the slab spectrum is assumed to follow
a Kolmochorov dependence in the inertial range and the
level of turbulence is assumed to be a fraction of the
large-scale background magnetic field. Alternatively, the
level of pitch-angle diffusion can be fixed by assuming that
the energetic particles have a constant radial or parallel
mean free path. Similarly, PARADISE includes different
perpendicular diffusion theories. The simplest model
assumes that the particles' have a constant perpendicular
mean free path. More advanced models, like e.g., the
non-linear guiding center theory (Matthaeus et al., 2003),
require the user to specify the properties of the 2D compo-
nent of the solar wind magnetic turbulence.

Apart from the diffusion theories, the particle injection
distribution introduces several free parameters in PARA-
DISE. In fact, the user has the freedom to provide its
own injection function with an associated set of free
parameters (e.g., Wijsen et al., 2019b). This particle injec-
tion distribution can, for example, depend on the MHD
variables of the EUHFORIA solar wind simulation (e.g.,
Wijsen et al., 2021; Wijsen et al., 2022). Other parameters
include the description of the injection energy spectrum,
which can be, for example, a mono-energetic or a (broken)
power law (e.g., Wijsen et al., 2020). The initial pitch-angle
distribution can also be specified to be (an) isotropic.

**Limitations and caveats:** When using the PARADISE model, the following limitations should be taken into account:

1. In PARADISE, the energetic particles are assumed to propagate as test particles through the solar wind. As a result, PARADISE does not capture the turbulence generated by energetic particles in the foreshock regions of CMEs, although this turbulence can be added in an ad hoc fashion (see Wijsen et al., 2022).

2. **PARADISE** provides a default particle injection function at CME-driven shock waves, which scales proportionally to the local divergence of the solar wind velocity. However, this injection function has not yet been calibrated to in situ data, which limits the capabilities of **PARADISE** as a SEP forecasting tool. In the future, this limitation will be addressed by using results from the SOLar Particle Acceleration in Coronal Shocks (SOLPACS; Afanasiev et al., 2015) model, which simulates particle acceleration at oblique shock waves, hereby taking into account the generation of Alfvén waves in the upstream shock region by the energetic particles.

**Access to model output or forecasts:** The PARADISE model is currently being implemented in the Virtual Space Weather Modelling Centre (VSWMC; Poedts et al., 2020) of the ESA. Once implemented, users will be able to perform PARADISE simulations through a front-end Graphical User Interface (GUI). In addition, the model repository is made available upon request.

**Model validation:** The validation of PARADISE is an actively ongoing process. The implementation of the model was first tested by performing simulations in a synthetic solar wind configuration containing a CIR (see Wijsen et al., 2019b; Wijsen et al., 2019a; Wijsen et al., 2020). More recently, the validation effort of the model has been focused on reproducing in situ observed energetic particle events. In Wijsen et al. (2021), PARADISE was used to successfully reproduce an energetic particle event associated with a CIR, which was observed in September 2019 by both PSP and STEREO-A. This simulation captured both the acceleration and the transport of the energetic protons associated to the CIR. In Wijsen et al. (2022), PARADISE was used to reproduce the ion intensity-time profiles observed near Earth on 14 July 2012. The CME generating this ESP event was simulated by using the spheromak model of EUHFORIA. For energies below ~1 MeV, the simulation results where shown to agree well with both the upstream and downstream components of the ESP event observed by ACE. In the future, we plan to extend our study by modelling several of the recently observed multi-spacecraft SEP events of SC25.

3.18. PCA (Papaioannou) Model - Principal Component Analysis

**Model developers:** Athanasios Papaioannou (NOA), Anastasios Anastasiadis (NOA), Athanasios Kouloum-vakos (Institut de Recherche en Astrophysique et Planetologie (IRAP)), Miikka Paassilta (University of Turku (UTU)), Rami Vainio (UTU), Eino Valtonen (UTU), Anatoly Belov (Pushkov Institute of Terrestrial Magnetism, Ionosphere and Radio Wave Propagation (IZMIRAN)), Eugenia Eroshenko (IZMIRAN), Maria Abunina (IZMIRAN), Artem Abunin (IZMIRAN).
The first three dimensions of the grid correspond to the
spatial dimensions and cover the inner heliosphere. The
two remaining dimensions of the grid correspond to the
energy and the pitch-angle coordinate of the energetic par-
ticles. The resolution of the output grid and the energy cov-
erage is defined by the user in the configuration file.

The model also provides the directional particle intensi-
ties, the first-order parallel anisotropies, and the omnidirec-
tional particle intensities at the location of the planets and
most spacecraft, including the STEREO spacecraft, PSP,
SO, and BepiColombo. In addition, the user can add addi-
tional virtual spacecraft to the simulations.

Finally, PARADISE contains python scripts and Jupy-
ter Notebooks which can be used to visualize the simula-
tion output.

**Free parameters:** PARADISE incorporates several dif-
ferent diffusion theories, which all have various free param-
eters. For example, the implemented pitch-angle diffusion
theories are derived from Quasi-Linear Theory (QLT;
Jokipii, 1966), and require the user to specify the properties
of the slab component of the solar wind magnetic turbu-
lence. As a default, the slab spectrum is assumed to follow
a Kolmochorov dependence in the inertial range and the
level of turbulence is assumed to be a fraction of the
large-scale background magnetic field. Alternatively, the
level of pitch-angle diffusion can be fixed by assuming that
the energetic particles have a constant radial or parallel
mean free path. Similarly, PARADISE includes different
perpendicular diffusion theories. The simplest model
assumes that the particles' have a constant perpendicular
mean free path. More advanced models, like e.g., the
non-linear guiding center theory (Matthaeus et al., 2003),
require the user to specify the properties of the 2D compo-
nent of the solar wind magnetic turbulence.

Apart from the diffusion theories, the particle injection
distribution introduces several free parameters in PARA-
DISE. In fact, the user has the freedom to provide its
own injection function with an associated set of free
parameters (e.g., Wijsen et al., 2019b). This particle injec-
tion distribution can, for example, depend on the MHD
variables of the EUHFORIA solar wind simulation (e.g.,
Wijsen et al., 2021; Wijsen et al., 2022). Other parameters
include the description of the injection energy spectrum,
which can be, for example, a mono-energetic or a (broken)
power law (e.g., Wijsen et al., 2020). The initial pitch-angle
distribution can also be specified to be (an) isotropic.

**Limitations and caveats:** When using the PARADISE model, the following limitations should be taken into account:

1. In PARADISE, the energetic particles are assumed to propagate as test particles through the solar wind. As a result, PARADISE does not capture the turbulence generated by energetic particles in the foreshock regions of CMEs, although this turbulence can be added in an ad hoc fashion (see Wijsen et al., 2022).

2. **PARADISE** provides a default particle injection function at CME-driven shock waves, which scales proportionally to the local divergence of the solar wind velocity. However, this injection function has not yet been calibrated to in situ data, which limits the capabilities of **PARADISE** as a SEP forecasting tool. In the future, this limitation will be addressed by using results from the SOLar Particle Acceleration in Coronal Shocks (SOLPACS; Afanasiev et al., 2015) model, which simulates particle acceleration at oblique shock waves, hereby taking into account the generation of Alfvén waves in the upstream shock region by the energetic particles.

**Access to model output or forecasts:** The PARADISE model is currently being implemented in the Virtual Space Weather Modelling Centre (VSWMC; Poedts et al., 2020) of the ESA. Once implemented, users will be able to perform PARADISE simulations through a front-end Graphical User Interface (GUI). In addition, the model repository is made available upon request.

**Model validation:** The validation of PARADISE is an actively ongoing process. The implementation of the model was first tested by performing simulations in a synthetic solar wind configuration containing a CIR (see Wijsen et al., 2019b; Wijsen et al., 2019a; Wijsen et al., 2020). More recently, the validation effort of the model has been focused on reproducing in situ observed energetic particle events. In Wijsen et al. (2021), PARADISE was used to successfully reproduce an energetic particle event associated with a CIR, which was observed in September 2019 by both PSP and STEREO-A. This simulation captured both the acceleration and the transport of the energetic protons associated to the CIR. In Wijsen et al. (2022), PARADISE was used to reproduce the ion intensity-time profiles observed near Earth on 14 July 2012. The CME generating this ESP event was simulated by using the spheromak model of EUHFORIA. For energies below ~1 MeV, the simulation results where shown to agree well with both the upstream and downstream components of the ESP event observed by ACE. In the future, we plan to extend our study by modelling several of the recently observed multi-spacecraft SEP events of SC25.

3.18. PCA (Papaioannou) Model - Principal Component Analysis

**Model developers:** Athanasios Papaioannou (NOA), Anastasios Anastasiadis (NOA), Athanasios Kouloum-vakos (Institut de Recherche en Astrophysique et Planetologie (IRAP)), Miikka Paassilta (University of Turku (UTU)), Rami Vainio (UTU), Eino Valtonen (UTU), Anatoly Belov (Pushkov Institute of Terrestrial Magnetism, Ionosphere and Radio Wave Propagation (IZMIRAN)), Eugenia Eroshenko (IZMIRAN), Maria Abunina (IZMIRAN), Artem Abunin (IZMIRAN).
**Model description:** The Papaioannou Model (Papaioannou et al., 2018) is a post-eruption model that uses Principal Component Analysis (PCA) to forecast the probability of SEP occurrence.

PCA is a dimension-reduction method typically used to reduce the number of correlated variables in large data sets. The method transforms the variables into orthogonal components called Principal Component (PC). The PCOMs are composed of uncorrelated variables in terms of the original correlated variables.

In the case of the Papaioannou Model, PCA is used to reduce the number of features and to create a combination of the uncorrelated features in order to calculate a probability of SEP occurrence. The features included here are the log of the flare peak flux, flare longitude, flare rise time, flare duration, CME velocity, and CME width. A sigmoid function is used to relate the combination of features to a probability.

$$P(x) = \frac{1}{1 - \exp[-\theta^T g(x)]} \quad (6)$$

Here, $g(x)$ is a function of the PCOMs, and $\theta^T$ are the coefficients. The function $\theta^T g(x)$ depends on the type of logistic regression used. The following are the types of regression considered:

* One-parameter linear: $\theta^T g(x) = \theta_0 + \theta_1 x$

* Multiple linear: $\theta^T g(x) = \theta_0 + \theta_1 x^{(1)} + \theta_2 x^{(2)}$

* Multiple non-linear: $\theta^T g(x) = \theta_0 + \theta_1 x^{(1)} + \theta_2 x^{(2)} + \theta_3 x^{(1)} x^{(2)}$

where $x$ can be any combination of PCOMs such as PCOM1, PCOM2, PCOM1 + PCOM2, etc. The coefficients were estimated by using the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method to reduce the negative log-likelihood function. This function reached a minimum for a probability threshold of 50% for event occurrence.

**Inputs:**

* Log of the flare peak flux

* Flare longitude

* Flare rise time

* Flare duration

* CME velocity

* CME width

**Outputs:**

* Probability of SEP occurrence

**Free Parameters:** None.

**Limitations and Caveats:** Model predictions are very straight forward. The model outputs the probability of SEP occurrence where an event is defined as >10 MeV protons above 10 pfu. A probability below 50% should be interpreted as no event will occur, while a probability above 50% should be interpreted as an event will occur.

**Access to model output or forecasts:** The PCA model is incorporated into FORSPEF. See Section 3.9 for access to forecasts.

**Model validation:** Validation included two methods: single variable logistic regression and multivariate logistic regression. Both methods used an optimized probability threshold (pt) of 50% for SEP occurrence. An event was defined as > 10 MeV protons above 10 pfu. For single variable logistic regression, 3598 predictions were made (34 true positive, 27 false positive, 3510 true negative, and 34 true positive). For multivariate logistic regression, 3663 predictions were made (69 true positives, 57 false negatives, 3507 false negatives, and 30 false positives). The table below summarizes the resulting POD, FAR, and HSS for all validations. The highest scoring method was a type of multivariate logistic regression with a POD of 58.73%, FAR of 24.49%, and HSS of 0.6502.

The best skill scores were I(3) and I(3 + O2) (see Table 7). As a result, their categorical measures were calculated as a function of the probability threshold. That is, the pt was treated as an independent parameter (not set to 50%) ranging between 0.0 to 1.0 with a step of 0.1. For both schemes, the optimal skill score was achieved at a range of pth from 25% to 40%. The optimal HSS is observed at pth = 0.33 (HSS = 0.6411) for I(3) and at pth = 0.25 (HSS = 0.6579) for I(3 + O2). In turn, this results in a POD = 65.87% and an FAR = 35.16% as well as a POD = 77.78% and an FAR = 40.96%, respectively.

<table><caption>Table 7
Results of validation from single variable and multivariate logistic regression applied in the Papaioannou (PCA) Model.</caption><thead><tr><th>index</th><th>Form (Scheme)</th><th>POD (%)</th><th>FAR (%)</th><th>HSS</th></tr></thead><tbody><tr><td>I<sup>(1)</sup></td><td>[PC1]</td><td>26.98</td><td>44.26</td><td>0.3490</td></tr><tr><td>I<sup>(2)</sup></td><td>[PC1,PC2]</td><td>54.76</td><td>30.30</td><td>0.6013</td></tr><tr><td>I<sup>(3)</sup></td><td>[PC1,PC2,PC3]</td><td>55.56</td><td>28.57</td><td>0.6134</td></tr><tr><td>I<sup>(4)</sup></td><td>[PC1,PC2,PC3,PC4]</td><td>53.97</td><td>29.17</td><td>0.6007</td></tr><tr><td>I<sup>(5)</sup></td><td>[PC1,...,PC5]</td><td>53.17</td><td>29.47</td><td>0.5943</td></tr><tr><td>I<sup>(6)</sup></td><td>[PC1,...,PC6]</td><td>53.17</td><td>28.72</td><td>0.5973</td></tr><tr><td>I<sup>(2+O<sup>2</sup>)</sup></td><td>[PC1,PC2,PC1<sup>2</sup>,PC2<sup>2</sup>,PC1-PC2]</td><td>56.35</td><td>31.07</td><td>0.6080</td></tr><tr><td>I<sup>(3+O<sup>2</sup>)</sup></td><td>[PC1,PC2,PC3,PC1<sup>2</sup>,PC2<sup>2</sup>,PC3<sup>2</sup>,PC1-PC2]</td><td>58.73</td><td>24.49</td><td>0.6502</td></tr></tbody></table>
**Model description:** The Papaioannou Model (Papaioannou et al., 2018) is a post-eruption model that uses Principal Component Analysis (PCA) to forecast the probability of SEP occurrence.

PCA is a dimension-reduction method typically used to reduce the number of correlated variables in large data sets. The method transforms the variables into orthogonal components called Principal Component (PC). The PCOMs are composed of uncorrelated variables in terms of the original correlated variables.

In the case of the Papaioannou Model, PCA is used to reduce the number of features and to create a combination of the uncorrelated features in order to calculate a probability of SEP occurrence. The features included here are the log of the flare peak flux, flare longitude, flare rise time, flare duration, CME velocity, and CME width. A sigmoid function is used to relate the combination of features to a probability.

$$P(x) = \frac{1}{1 - \exp[-\theta^T g(x)]} \quad (6)$$

Here, $g(x)$ is a function of the PCOMs, and $\theta^T$ are the coefficients. The function $\theta^T g(x)$ depends on the type of logistic regression used. The following are the types of regression considered:

* One-parameter linear: $\theta^T g(x) = \theta_0 + \theta_1 x$

* Multiple linear: $\theta^T g(x) = \theta_0 + \theta_1 x^{(1)} + \theta_2 x^{(2)}$

* Multiple non-linear: $\theta^T g(x) = \theta_0 + \theta_1 x^{(1)} + \theta_2 x^{(2)} + \theta_3 x^{(1)} x^{(2)}$

where $x$ can be any combination of PCOMs such as PCOM1, PCOM2, PCOM1 + PCOM2, etc. The coefficients were estimated by using the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method to reduce the negative log-likelihood function. This function reached a minimum for a probability threshold of 50% for event occurrence.

**Inputs:**

* Log of the flare peak flux

* Flare longitude

* Flare rise time

* Flare duration

* CME velocity

* CME width

**Outputs:**

* Probability of SEP occurrence

**Free Parameters:** None.

**Limitations and Caveats:** Model predictions are very straight forward. The model outputs the probability of SEP occurrence where an event is defined as >10 MeV protons above 10 pfu. A probability below 50% should be interpreted as no event will occur, while a probability above 50% should be interpreted as an event will occur.

**Access to model output or forecasts:** The PCA model is incorporated into FORSPEF. See Section 3.9 for access to forecasts.

**Model validation:** Validation included two methods: single variable logistic regression and multivariate logistic regression. Both methods used an optimized probability threshold (pt) of 50% for SEP occurrence. An event was defined as > 10 MeV protons above 10 pfu. For single variable logistic regression, 3598 predictions were made (34 true positive, 27 false positive, 3510 true negative, and 34 true positive). For multivariate logistic regression, 3663 predictions were made (69 true positives, 57 false negatives, 3507 false negatives, and 30 false positives). The table below summarizes the resulting POD, FAR, and HSS for all validations. The highest scoring method was a type of multivariate logistic regression with a POD of 58.73%, FAR of 24.49%, and HSS of 0.6502.

The best skill scores were $I(3)$ and $I(3 + O_2)$ (see Table 7). As a result, their categorical measures were calculated as a function of the probability threshold. That is, the pt was treated as an independent parameter (not set to 50%) ranging between 0.0 to 1.0 with a step of 0.1. For both schemes, the optimal skill score was achieved at a range of pth from 25% to 40%. The optimal HSS is observed at pth = 0.33 (HSS = 0.6411) for $I(3)$ and at pth = 0.25 (HSS = 0.6579) for $I(3 + O_2)$. In turn, this results in a POD = 65.87% and an FAR = 35.16% as well as a POD = 77.78% and an FAR = 40.96%, respectively.

<table><caption>Table 7
Results of validation from single variable and multivariate logistic regression applied in the Papaioannou (PCA) Model.</caption><thead><tr><th>index</th><th>Form (Scheme)</th><th>POD (%)</th><th>FAR (%)</th><th>HSS</th></tr></thead><tbody><tr><td>$I^{(1)}$</td><td>[PC1]</td><td>26.98</td><td>44.26</td><td>0.3490</td></tr><tr><td>$I^{(2)}$</td><td>[PC1,PC2]</td><td>54.76</td><td>30.30</td><td>0.6013</td></tr><tr><td>$I^{(3)}$</td><td>[PC1,PC2,PC3]</td><td>55.56</td><td>28.57</td><td>0.6134</td></tr><tr><td>$I^{(4)}$</td><td>[PC1,PC2,PC3,PC4]</td><td>53.97</td><td>29.17</td><td>0.6007</td></tr><tr><td>$I^{(5)}$</td><td>[PC1,...,PC5]</td><td>53.17</td><td>29.47</td><td>0.5943</td></tr><tr><td>$I^{(6)}$</td><td>[PC1,...,PC6]</td><td>53.17</td><td>28.72</td><td>0.5973</td></tr><tr><td>$I^{(2+O_2)}$</td><td>[PC1,PC2,PC1<sup>2</sup>,PC2<sup>2</sup>,PC1·PC2]</td><td>56.35</td><td>31.07</td><td>0.6080</td></tr><tr><td>$I^{(3+O_2)}$</td><td>[PC1,PC2,PC3,PC1<sup>2</sup>,PC2<sup>2</sup>,PC3<sup>2</sup>,PC1·PC2]</td><td>58.73</td><td>24.49</td><td>0.6502</td></tr></tbody></table>
## 3.19. PHSVM

**Model developers:** Pouya Hosseinzadeh, Soukaina Filali Boubrahimi (Utah State University).

**Model description:** Support Vector Machine (SVM) is a machine learning model that finds a hyperplane that best divides the dataset into different classes. PHSVM is an SVM model that follows the same idea described in Boubrahimi et al. (2017), in order to predict >100 MeV SEP events and non-SEP events.

The data used to train PHSVM is composed of GOES X-ray flux and proton flux channels as multivariate time series that led to >100 MeV events, considered as the positive class, and multivariate time series proton data that did not lead to >100 MeV events, which are considered as the negative class. Specifically, PHSVM considers X-ray channel (xl) and proton channel P8 (p8_flux) as a multivariate time series to classify specific events into SEP and non-SEP events.

One of the model's hyperparameters is the span window, which is defined as the number of hours that constitute the observation period prior to an X-ray event. A span of 120 time steps has been found to be optimal for training our model. Each time step corresponds to a 5-min average of X-ray or proton flux channels, which makes our span equivalent to 10 h. PHSVM considers each SEP or non-SEP sample as a 2x120 matrix that is composed of 120 X-ray and 120 proton flux channel data points.

The input training data for training the model is composed of a balanced number of samples, 50 (2 × 25) samples for the positive and negative classes that are collected from GOES measurements from the period between 1997 and 2002.

**Inputs:** The model inputs are multivariate times series of two channels from the GOES X-ray and proton instruments.

**Outputs:** The model output is a binary flag for >100 MeV SEP event or non-SEP event.

**Free parameters:** This is a machine learning model and it does not contain free parameters in the same sense as a physical model. However, the span window and SVM kernel are set during the model's hyper-parameter search.

**Limitations and caveats:** This is an “all clear” evaluation indicating whether there is >100 MeV SEP event or non-SEP events. Therefore, predicting the exact intensity is missing since other energy channels are ignored. Furthermore, the training and validation sets used in this study are not adequate for providing a general and robust result. One may consider more events for training and testing the model.

**Access to model output or forecasts:** Output from this model is not currently available to the public.

**Model validation:** The model was trained on 50 events including both classes. The evaluation process was performed on 10 non-SEP and 6 SEP events. The result was as follows: accuracy = 0.86, recall = 0.83, precision = 0.83, F1 = 0.83. Furthermore, we added a lag time of 5 h before

the solar flare onset time to provide the stakeholders with extra time to make decisions. The accuracy dropped to 0.83.

## 3.20. PROTONS

**Model developers:** Christopher C. Balch (NOAA SWPC).

**Model description:** The PROTONS model is the operational SEP proton prediction model currently used by the NOAA SWPC forecast office (Balch, 1999; Balch, 2008). It predicts >10 MeV SEP occurrence and maximum flux from ≥C2.4 flares derived from time-integrated SXR flux, peak SXR flux, the occurrence or non-occurrence of metric radio type II and type IV sweeps, and the location of the flare.

The probability prediction is based on the GOES XRS maximum X-ray flux, the X-ray flux integrated from the flare onset until the flux decays to half of the maximum flux value, and the occurrence of Type II or Type IV radio bursts reported from the USAF Solar Electro-Optical Observatory Network (SEON). The probability model was trained with 88 proton events and 1334 control events between 1986 to 2004. These events were classified into groups defined by integrated X-ray flux, maximum X-ray flux, and radio sweep values, generating a 5 x 5 x 4 parameter space. For each point in parameter space, the number of proton events was divided by the total number of events to estimate the probability of proton event occurrence. One dimension of the parameter space was removed as necessary when the sample size for a particular point in parameter space was too small (Balch, 1999; Balch, 2008).

The proton maximum flux model is based primarily on a statistical relationship between the log of the >10 MeV peak flux of the proton events and the log of the integrated X-ray flux of the associated X-ray event (Balch, 2008). The prediction is also informed by the integrated X-ray flux of the most recent previous event that occurred in the same active region. The prediction is given by the formula:

$$ \Phi_{P10} = 10\alpha_{PF} \times \left[ \frac{X_{int}}{0.00987} \right]^{0.82} \quad (7) $$

where $\Phi_{P10}$ is the predicted maximum flux for >10 MeV, $X_{int}$ is the integrated X-ray flux for the associated flare, and

$$ \alpha_{PF} = \begin{cases} \left( \frac{X_{pfint}}{0.167} \right)^{1.146} & \text{if } X_{pfint} > 0.08, \\ 1 & \text{if } X_{pfint} \le 0.08 \end{cases} \quad (8) $$

where $X_{pfint}$ is the integrated X-ray flux of the previous flare event.

The rise time is defined as the time difference between the X-ray maximum and proton maximum. The prediction of this value is derived from the location of the associated flare on the solar disk using the formula:
## 3.19. PHSVM

**Model developers:** Pouya Hosseinzadeh, Soukaina Filali Boubrahimi (Utah State University).

**Model description:** Support Vector Machine (SVM) is a machine learning model that finds a hyperplane that best divides the dataset into different classes. PHSVM is an SVM model that follows the same idea described in Boubrahimi et al. (2017), in order to predict >100 MeV SEP events and non-SEP events.

The data used to train PHSVM is composed of GOES X-ray flux and proton flux channels as multivariate time series that led to >100 MeV events, considered as the positive class, and multivariate time series proton data that did not lead to >100 MeV events, which are considered as the negative class. Specifically, PHSVM considers X-ray channel (xl) and proton channel P8 (p8_flux) as a multivariate time series to classify specific events into SEP and non-SEP events.

One of the model's hyperparameters is the span window, which is defined as the number of hours that constitute the observation period prior to an X-ray event. A span of 120 time steps has been found to be optimal for training our model. Each time step corresponds to a 5-min average of X-ray or proton flux channels, which makes our span equivalent to 10 h. PHSVM considers each SEP or non-SEP sample as a 2x120 matrix that is composed of 120 X-ray and 120 proton flux channel data points.

The input training data for training the model is composed of a balanced number of samples, 50 (2 × 25) samples for the positive and negative classes that are collected from GOES measurements from the period between 1997 and 2002.

**Inputs:** The model inputs are multivariate times series of two channels from the GOES X-ray and proton instruments.

**Outputs:** The model output is a binary flag for >100 MeV SEP event or non-SEP event.

**Free parameters:** This is a machine learning model and it does not contain free parameters in the same sense as a physical model. However, the span window and SVM kernel are set during the model's hyper-parameter search.

**Limitations and caveats:** This is an “all clear” evaluation indicating whether there is >100 MeV SEP event or non-SEP events. Therefore, predicting the exact intensity is missing since other energy channels are ignored. Furthermore, the training and validation sets used in this study are not adequate for providing a general and robust result. One may consider more events for training and testing the model.

**Access to model output or forecasts:** Output from this model is not currently available to the public.

**Model validation:** The model was trained on 50 events including both classes. The evaluation process was performed on 10 non-SEP and 6 SEP events. The result was as follows: accuracy = 0.86, recall = 0.83, precision = 0.83, F1 = 0.83. Furthermore, we added a lag time of 5 h before

the solar flare onset time to provide the stakeholders with extra time to make decisions. The accuracy dropped to 0.83.

## 3.20. PROTONS

**Model developers:** Christopher C. Balch (NOAA SWPC).

**Model description:** The PROTONS model is the operational SEP proton prediction model currently used by the NOAA SWPC forecast office (Balch, 1999; Balch, 2008). It predicts >10 MeV SEP occurrence and maximum flux from ≥C2.4 flares derived from time-integrated SXR flux, peak SXR flux, the occurrence or non-occurrence of metric radio type II and type IV sweeps, and the location of the flare.

The probability prediction is based on the GOES XRS maximum X-ray flux, the X-ray flux integrated from the flare onset until the flux decays to half of the maximum flux value, and the occurrence of Type II or Type IV radio bursts reported from the USAF Solar Electro-Optical Observatory Network (SEON). The probability model was trained with 88 proton events and 1334 control events between 1986 to 2004. These events were classified into groups defined by integrated X-ray flux, maximum X-ray flux, and radio sweep values, generating a 5 x 5 x 4 parameter space. For each point in parameter space, the number of proton events was divided by the total number of events to estimate the probability of proton event occurrence. One dimension of the parameter space was removed as necessary when the sample size for a particular point in parameter space was too small (Balch, 1999; Balch, 2008).

The proton maximum flux model is based primarily on a statistical relationship between the log of the >10 MeV peak flux of the proton events and the log of the integrated X-ray flux of the associated X-ray event (Balch, 2008). The prediction is also informed by the integrated X-ray flux of the most recent previous event that occurred in the same active region. The prediction is given by the formula:

$$ \Phi_{P10} = 10\alpha_{PF} \times \left[ \frac{X_{int}}{0.00987} \right]^{0.82} \quad (7) $$

where $\Phi_{P10}$ is the predicted maximum flux for >10 MeV, $X_{int}$ is the integrated X-ray flux for the associated flare, and

$$ \alpha_{PF} = \begin{cases} \left( \frac{X_{pfint}}{0.167} \right)^{1.146} & \text{if } X_{pfint} > 0.08, \\ 1 & \text{if } X_{pfint} \le 0.08 \end{cases} \quad (8) $$

where $X_{pfint}$ is the integrated X-ray flux of the previous flare event.

The rise time is defined as the time difference between the X-ray maximum and proton maximum. The prediction of this value is derived from the location of the associated flare on the solar disk using the formula:
$$t_{rise} = t_{min} + \left[ \frac{\text{longitude} - l_{SE}}{\lambda} \right]^2 \qquad (9)$$

where $l_{SE}$ is the optimal sub-Earth longitude (78 degrees west), $\lambda$ is the longitudinal scaling factor (18.1 degrees), and $t_{min}$ is the minimum rise time (9.4 h).

### Inputs:

* Peak SXR flux

* Time-integrated SXR flux

* Flare location

* Occurrence or non-occurrence of metric Type II and Type IV radio bursts.

### Outputs:

* Probability of proton event occurrence

* Maximum proton flux

* Rise time (time difference between the X-ray max and proton max)

**Free parameters:** None.

**Limitations and Caveats:** An analysis found the model overforecasts in the 0.55–0.65 probability range and underforecasts in the 0.25–0.35 probability range (Balch, 2008).

**Access to model output or forecasts:** This model is used internally by NOAA SWPC and is not available to the public.

**Model validation:** Balch (2008) describes the validation of the model along with an explanation of each of the metrics used. The validation was extensive and a selected set of results are included here.

The proton data set was rederived with corrections applied to GOES data prior to 1990, resulting in a list of 165 events from 1986–2004. Instead of strictly using the >10 MeV, 10 pfu definition, cases where an “event” was due to multiple injections in a row, each injection was treated as a separate SEP event and associated with its flare source.

For their analysis, flares occurring behind the limb automatically counted as a miss. About 17% of the source originated behind the west limb and two events were behind the east limb. 5.4% (9) qualified as events in the ESP phase. These events were treated as a priori missed events. These excluded events places an upper limit on the probability of detection at 77%.

For the remaining 127 events, analysis of the event parameters showed that all were associated with an XRS event with peak flux in the 1–8 Åband of greater than 2.44 × 10⁻⁶ Watts m⁻², integrated X-ray flux greater than 9.93 × 10⁻³ Joules m⁻², and background-subtracted integrated X-ray flux greater than 5.95 × 10⁻³ Joules m⁻². All XRS events that satisfied these conditions but did not produce a proton event resulted in a list of 3656 negative events from 1986 to 2004.

They presented a forecast reliability diagram and skill scores calculated by applying varying thresholds to convert the probabilistic score to a binary forecast. The optimum threshold of 20–30% probability results in a POD of 0.57, FAR of 0.55, PCOR of 96%, and HSS of 0.48. For the peak flux forecast, the correlation coefficient was 0.524 and the mean absolute log error for peak flux forecast was 0.87. The mean error for rise time prediction was 12.6 h.

## 3.21. REleASE and HESPERIA/REleASE - Relativistic Electron Alert system for Exploration

**Model developers:** Arik Posner (NASA HQ), Olga E. Malandraki (NOA), Bernd Heber (Institute of Experimental and Applied Physics (IEAP)), Johannes Labrenz (IEAP), Patrick Kühl (IEAP), R. Du Toit Strauss (North-West University).

Initial development of the RELEASE model is described in Posner (2007). Further development of what is now called the HESPERIA/RELEASE model has been performed in the context of the HESPERIA HORIZON 2020 EU program, as described in Malandraki and Crosby (2018)

**Model description:** The current High Energy Solar Particle Events foRecastIng and Analysis (HESPERIA)/Relativistic Electron Alert System for Exploration (REleASE) model takes advantage of promptly arriving near-relativistic as well as relativistic electrons as an early indicator of a solar energetic particle event. It uses 175–315 keV electron measurements from ACE/Electron, Proton, and Alpha Monitor (EPAM) and 0.25–1 MeV electrons measured by SOHO/Comprehensive Suprathermal and Energetic Particle Analyzer (COSTEP) to predict 15.6–39.8 MeV and 28.2–50.1 MeV proton fluxes in the 30, 60 and 90 min future windows. Relativistic electrons ~1 MeV always arrive at 1 AU prior to the ~30–100 MeV protons (Posner, 2007). For a nominal Parker Spiral of 1.2 AU from the Sun to the Earth, 1 MeV electrons would arrive 30.4 min earlier than 30 MeV protons, 21.8 min earlier than 50 MeV protons, and 13.3 min ahead of 100 MeV protons, but would lag behind 300 MeV protons by more than 5 min (Posner, 2007). While this method does not provide early warning for the highest energy SEP protons arriving at Earth, it does provide advance warning for the bulk of SEP events as well as acts as an indicator of magnetic connectivity between the Sun and Earth, i.e., if energetic electrons arrive at Earth, it is clear that a magnetic connection exists between the Earth and active region and energetic protons are expected. A unique aspect of this forecast method is inherent in the use of in situ particle measurements (i.e., electrons at L1) as opposed to reliance on remote solar observations (e.g., X-rays). HESPERIA/RELEASE can therefore forecast SEP events associated with solar eruptions behind the western solar limb, an expansion of capabilities beyond those offered by models relying on remote sensing observations only available for the visible solar disc.
$$t_{rise} = t_{min} + \left[ \frac{\text{longitude} - l_{SE}}{\lambda} \right]^2 \quad (9)$$

where $l_{SE}$ is the optimal sub-Earth longitude (78 degrees west), $\lambda$ is the longitudinal scaling factor (18.1 degrees), and $t_{min}$ is the minimum rise time (9.4 h).

### Inputs:

* Peak SXR flux

* Time-integrated SXR flux

* Flare location

* Occurrence or non-occurrence of metric Type II and Type IV radio bursts.

### Outputs:

* Probability of proton event occurrence

* Maximum proton flux

* Rise time (time difference between the X-ray max and proton max)

**Free parameters:** None.

**Limitations and Caveats:** An analysis found the model overforecasts in the 0.55–0.65 probability range and underforecasts in the 0.25–0.35 probability range (Balch, 2008).

**Access to model output or forecasts:** This model is used internally by NOAA SWPC and is not available to the public.

**Model validation:** Balch (2008) describes the validation of the model along with an explanation of each of the metrics used. The validation was extensive and a selected set of results are included here.

The proton data set was rederived with corrections applied to GOES data prior to 1990, resulting in a list of 165 events from 1986–2004. Instead of strictly using the >10 MeV, 10 pfu definition, cases where an “event” was due to multiple injections in a row, each injection was treated as a separate SEP event and associated with its flare source.

For their analysis, flares occurring behind the limb automatically counted as a miss. About 17% of the source originated behind the west limb and two events were behind the east limb. 5.4% (9) qualified as events in the ESP phase. These events were treated as a priori missed events. These excluded events places an upper limit on the probability of detection at 77%.

For the remaining 127 events, analysis of the event parameters showed that all were associated with an XRS event with peak flux in the 1–8 Åband of greater than 2.44 × 10⁻⁶ Watts m⁻², integrated X-ray flux greater than 9.93 × 10⁻³ Joules m⁻², and background-subtracted integrated X-ray flux greater than 5.95 × 10⁻³ Joules m⁻². All XRS events that satisfied these conditions but did not produce a proton event resulted in a list of 3656 negative events from 1986 to 2004.

They presented a forecast reliability diagram and skill scores calculated by applying varying thresholds to convert the probabilistic score to a binary forecast. The optimum threshold of 20–30% probability results in a POD of 0.57, FAR of 0.55, PCOR of 96%, and HSS of 0.48. For the peak flux forecast, the correlation coefficient was 0.524 and the mean absolute log error for peak flux forecast was 0.87. The mean error for rise time prediction was 12.6 h.

## 3.21. REleASE and HESPERIA/REleASE - Relativistic Electron Alert system for Exploration

**Model developers:** Arik Posner (NASA HQ), Olga E. Malandraki (NOA), Bernd Heber (Institute of Experimental and Applied Physics (IEAP)), Johannes Labrenz (IEAP), Patrick Kühl (IEAP), R. Du Toit Strauss (North-West University).

Initial development of the RELEASE model is described in Posner (2007). Further development of what is now called the HESPERIA/RELEASE model has been performed in the context of the HESPERIA HORIZON 2020 EU program, as described in Malandraki and Crosby (2018)

**Model description:** The current High Energy Solar Particle Events foRecastIng and Analysis (HESPERIA)/Relativistic Electron Alert System for Exploration (REleASE) model takes advantage of promptly arriving near-relativistic as well as relativistic electrons as an early indicator of a solar energetic particle event. It uses 175–315 keV electron measurements from ACE/Electron, Proton, and Alpha Monitor (EPAM) and 0.25–1 MeV electrons measured by SOHO/Comprehensive Suprathermal and Energetic Particle Analyzer (COSTEP) to predict 15.6–39.8 MeV and 28.2–50.1 MeV proton fluxes in the 30, 60 and 90 min future windows. Relativistic electrons ~1 MeV always arrive at 1 AU prior to the ~30–100 MeV protons (Posner, 2007). For a nominal Parker Spiral of 1.2 AU from the Sun to the Earth, 1 MeV electrons would arrive 30.4 min earlier than 30 MeV protons, 21.8 min earlier than 50 MeV protons, and 13.3 min ahead of 100 MeV protons, but would lag behind 300 MeV protons by more than 5 min (Posner, 2007). While this method does not provide early warning for the highest energy SEP protons arriving at Earth, it does provide advance warning for the bulk of SEP events as well as acts as an indicator of magnetic connectivity between the Sun and Earth, i.e., if energetic electrons arrive at Earth, it is clear that a magnetic connection exists between the Earth and active region and energetic protons are expected. A unique aspect of this forecast method is inherent in the use of in situ particle measurements (i.e., electrons at L1) as opposed to reliance on remote solar observations (e.g., X-rays). HESPERIA/RELEASE can therefore forecast SEP events associated with solar eruptions behind the western solar limb, an expansion of capabilities beyond those offered by models relying on remote sensing observations only available for the visible solar disc.
The RELEASE project combined data and knowledge from 9 European partners and several collaborating parties from the US and Russia. The HESPERIA project produced two novel forecasting tools based upon proven concepts: HESPERIA/UMASEP-500 (Section 3.35) and HESPERIA/RELEASE. The RELEASE model uses the relationship between the propagation speeds of relativistic electrons and ions during transport from the sun to 1 AU. This could plausibly be extended to other locations for the observer, such as Mars. The relativistic electrons from Solar Particle Event (SPE) s, highly abundant and easy to detect outside the magnetosphere, can be used as messengers for the later arriving ions (predominantly protons) energetic enough to affect humans and technology in space. Superposed epoch and statistical analyses revealed that 1) the behaviors of the electron and proton increase parameters are very similar and 2) the maximum flux intensity between protons and electrons has a clear ordering over a broad range of SPEs. It is therefore feasible to forecast proton fluxes based on electron onset observations.

A list of solar proton events, defined as >10 MeV exceeds 10 pfu, measured by GOES-8 from 1996–2002 serves as a basis to derive the fundamentals of the forecasting method with statistical and superposed epoch techniques. To derive the intensity relationship between electrons and protons during event onset, the slope of the rise in electron flux, called the electron increase parameter, and the slope in the rise of the proton flux were derived. The authors found that the rise characteristics were similar for both electrons and protons. Making use of this information, a forecasting matrix was generated with the electron increase parameter on one axis and electron intensity on the other axis. For each element in the matrix, the predicted proton flux was derived (See Figure 8 in Posner (2007)).

HESPERIA/RELEASE also accounts for the delay between electron and proton arrival at 1 AU. It was found that the average time delay between the arrival of electrons and <50 MeV protons was 63.3 min. Due to varying magnetic connectivity conditions, most protons arrived within 30–90 min of electron arrival. To account for the possible magnetic connectivity of any given SEP event, the model issues three separate predictions of exactly the same proton fluxes, but with 30-, 60-, and 90-min delays applied. These forecasts can be considered to represent the range of possible proton arrivals at Earth resulting from the most statistically likely connectivities. Which of the three turns out to be the most accurate forecast will depend on the magnetic connectivity of the specific event. The current HESPERIA/RELEASE version builds on the work of the original RELEASE model, relying on the created empirical matrices to develop a prediction 30, 60, and 90 min into the future. The matrices are based on historic measurements of proton flux, observed 30, 60, and 90 min after the electrons, as a function of both the electron flux and the slope of the electron flux increase.

HESPERIA/RELEASE issues a warning if the following conditions are met:

* The average proton flux for the previous two hours is below the hazard level (20 pfu)

* The predicted proton flux exceeds the hazard level by 50% (30 pfu)

* The predicted value at the time of the warning is required to be the maximum predicted value over the previous 2 h

Fig. 7 illustrates the HESPERIA REleASE tool’s panel with the forecasting results for the successfully predicted SEP event on the 28th of March 2022. The upper panel on both sides shows real time proton fluxes measured by SOHO/EPHIN. The proton flux in the energy range of 15.8 to 39.8 MeV is shown in red and the proton flux in the energy range of 28.2 to 50.1 MeV is shown in green. The next three panels show the proton flux forecasts obtained with the HESPERIA REleASE forecasting scheme. From top to bottom the panels show expected proton fluxes in 30, 60, and 90 min (time-shifted). On the left and right sides, the forecasts are based on electron measurements by SOHO/EPHIN and ACE/EPAM, respectively. All forecasts use the same color code as the real-time proton flux in the top panel. The bottom panels on both sides show the measured real-time electron fluxes used for the forecasts. The SOHO/EPHIN and ACE/EPAM electron fluxes cover the energy range of 0.25 to 1.0 MeV and 0.175 to 0.315 MeV, respectively. Also, the first dashed line at 0.1 flux/ (cm² s sr MeV)⁻¹, is the HESPERIA REleASE proton flux intensity threshold. The second dashed line is shown to guide the eye of the user about the peak intensity of the event and whether this crosses the threshold of 10 flux/ (cm² s sr MeV)⁻¹. Furthermore, the vertical grey shaded areas indicate the time periods during which the electron input data for the ACE/EPAM and SOHO/EPHIN experiments is available, respectively. It can be observed that the ACE/EPAM provides full coverage concerning the electron input data for the time period where the SEP event occurred.

Fig. 8 presents the actual SOHO COSTEP/EPHIN proton fluxes observations (blue curve) along with the forecasted proton fluxes in 60 min, produced by the HESPERIA REleASE tool utilizing the ACE/EPAM electron input data (red curve). The actual and the forecasted proton fluxes are assessed for the energy range of 28.2 to 50.1 MeV, essentially the critical energy range for human spaceflight. Also, to determine the advanced warning time (AWT), note that there is no time shift applied to the 60 min forecast and the red points are plotted at the time that the forecast was issued. The red curve, i.e., the forecasted proton fluxes produced by the HESPERIA REleASE scheme, crosses the threshold on 12:00:00 and generates a proton alert, while from the actual measured proton fluxes (blue curve) the threshold is crossed on 13:53:02. This gives an important AWT of 113.03 min, concerning the SEP
The RELEASE project combined data and knowledge from 9 European partners and several collaborating parties from the US and Russia. The HESPERIA project produced two novel forecasting tools based upon proven concepts: HESPERIA/UMASEP-500 (Section 3.35) and HESPERIA/RELEASE. The RELEASE model uses the relationship between the propagation speeds of relativistic electrons and ions during transport from the sun to 1 AU. This could plausibly be extended to other locations for the observer, such as Mars. The relativistic electrons from Solar Particle Event (SPE) s, highly abundant and easy to detect outside the magnetosphere, can be used as messengers for the later arriving ions (predominantly protons) energetic enough to affect humans and technology in space. Superposed epoch and statistical analyses revealed that 1) the behaviors of the electron and proton increase parameters are very similar and 2) the maximum flux intensity between protons and electrons has a clear ordering over a broad range of SPEs. It is therefore feasible to forecast proton fluxes based on electron onset observations.

A list of solar proton events, defined as >10 MeV exceeds 10 pfu, measured by GOES-8 from 1996–2002 serves as a basis to derive the fundamentals of the forecasting method with statistical and superposed epoch techniques. To derive the intensity relationship between electrons and protons during event onset, the slope of the rise in electron flux, called the electron increase parameter, and the slope in the rise of the proton flux were derived. The authors found that the rise characteristics were similar for both electrons and protons. Making use of this information, a forecasting matrix was generated with the electron increase parameter on one axis and electron intensity on the other axis. For each element in the matrix, the predicted proton flux was derived (See Figure 8 in Posner (2007)).

HESPERIA/RELEASE also accounts for the delay between electron and proton arrival at 1 AU. It was found that the average time delay between the arrival of electrons and <50 MeV protons was 63.3 min. Due to varying magnetic connectivity conditions, most protons arrived within 30–90 min of electron arrival. To account for the possible magnetic connectivity of any given SEP event, the model issues three separate predictions of exactly the same proton fluxes, but with 30-, 60-, and 90-min delays applied. These forecasts can be considered to represent the range of possible proton arrivals at Earth resulting from the most statistically likely connectivities. Which of the three turns out to be the most accurate forecast will depend on the magnetic connectivity of the specific event. The current HESPERIA/RELEASE version builds on the work of the original RELEASE model, relying on the created empirical matrices to develop a prediction 30, 60, and 90 min into the future. The matrices are based on historic measurements of proton flux, observed 30, 60, and 90 min after the electrons, as a function of both the electron flux and the slope of the electron flux increase.

HESPERIA/RELEASE issues a warning if the following conditions are met:

* The average proton flux for the previous two hours is below the hazard level (20 pfu)

* The predicted proton flux exceeds the hazard level by 50% (30 pfu)

* The predicted value at the time of the warning is required to be the maximum predicted value over the previous 2 h

Fig. 7 illustrates the HESPERIA REleASE tool's panel with the forecasting results for the successfully predicted SEP event on the 28th of March 2022. The upper panel on both sides shows real time proton fluxes measured by SOHO/EPHIN. The proton flux in the energy range of 15.8 to 39.8 MeV is shown in red and the proton flux in the energy range of 28.2 to 50.1 MeV is shown in green. The next three panels show the proton flux forecasts obtained with the HESPERIA REleASE forecasting scheme. From top to bottom the panels show expected proton fluxes in 30, 60, and 90 min (time-shifted). On the left and right sides, the forecasts are based on electron measurements by SOHO/EPHIN and ACE/EPAM, respectively. All forecasts use the same color code as the real-time proton flux in the top panel. The bottom panels on both sides show the measured real-time electron fluxes used for the forecasts. The SOHO/EPHIN and ACE/EPAM electron fluxes cover the energy range of 0.25 to 1.0 MeV and 0.175 to 0.315 MeV, respectively. Also, the first dashed line at 0.1 flux/ (cm² s sr MeV)⁻¹, is the HESPERIA REleASE proton flux intensity threshold. The second dashed line is shown to guide the eye of the user about the peak intensity of the event and whether this crosses the threshold of 10 flux/ (cm² s sr MeV)⁻¹. Furthermore, the vertical grey shaded areas indicate the time periods during which the electron input data for the ACE/EPAM and SOHO/EPHIN experiments is available, respectively. It can be observed that the ACE/EPAM provides full coverage concerning the electron input data for the time period where the SEP event occurred.

Fig. 8 presents the actual SOHO COSTEP/EPHIN proton fluxes observations (blue curve) along with the forecasted proton fluxes in 60 min, produced by the HESPERIA REleASE tool utilizing the ACE/EPAM electron input data (red curve). The actual and the forecasted proton fluxes are assessed for the energy range of 28.2 to 50.1 MeV, essentially the critical energy range for human spaceflight. Also, to determine the advanced warning time (AWT), note that there is no time shift applied to the 60 min forecast and the red points are plotted at the time that the forecast was issued. The red curve, i.e., the forecasted proton fluxes produced by the HESPERIA REleASE scheme, crosses the threshold on 12:00:00 and generates a proton alert, while from the actual measured proton fluxes (blue curve) the threshold is crossed on 13:53:02. This gives an important AWT of 113.03 min, concerning the SEP
event. Therefore, astronauts performing EVA activities
have ample time to take shelter.

**Inputs:**

Original Version:

* Energetic electron measurements from COTSEP

HESPERIA Version:

* Relativistic electron (v > 0.9c) measurements from SOHO/ COTSEP/ EPHIN

* Near-relativistic electron (v < 0.8c) measurements from ACE/ EPAM

**Outputs:**

Original Version:

* Proton fluxes at the following four COTSEP energy bands to facilitate direct model and data comparison/performance evaluation: 4–9 MeV; 9–15.8 MeV; 15.8–39.8 MeV; 28.2–50.1 MeV.

* Proton flux alerts at 15.8–39.8 MeV and 28.2–50.1 MeV.

HESPERIA Version:

* Proton flux alerts at 15.8–39.8 MeV and 28.2–50.1 MeV (COTSEP input).

* Proton flux alerts at 15.8–39.8 MeV and 28.2–50.1 MeV (EPAM input).

**Free parameters:** None.

**Limitations and caveats:** HESPERIA/RELEASE relies on data availability and measurement quality. Current detectors are not ideal for predicting extreme events.

<figure><img src="image_8.png" /><figcaption>Fig. 7. HESPERIA REleASE results for the SEP event on 28 March 2022.</figcaption></figure>
event. Therefore, astronauts performing EVA activities
have ample time to take shelter.

**Inputs:**

Original Version:

* Energetic electron measurements from COTSEP

HESPERIA Version:

* Relativistic electron (v > 0.9c) measurements from SOHO/ COTSEP/ EPHIN

* Near-relativistic electron (v < 0.8c) measurements from ACE/ EPAM

**Outputs:**

Original Version:

* Proton fluxes at the following four COTSEP energy bands to facilitate direct model and data comparison/performance evaluation: 4–9 MeV; 9–15.8 MeV; 15.8–39.8 MeV; 28.2–50.1 MeV.

* Proton flux alerts at 15.8–39.8 MeV and 28.2–50.1 MeV.

HESPERIA Version:

* Proton flux alerts at 15.8–39.8 MeV and 28.2–50.1 MeV (COTSEP input).

* Proton flux alerts at 15.8–39.8 MeV and 28.2–50.1 MeV (EPAM input).

**Free parameters:** None.

**Limitations and caveats:** HESPERIA/RELEASE relies on data availability and measurement quality. Current detectors are not ideal for predicting extreme events.

<figure><img src="image_8.png" /><figcaption>Fig. 7. HESPERIA REleASE results for the SEP event on 28 March 2022.</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>Fig. 8. HESPERIA REleASE AWT for the SEP event on the 28th of March 2022.</figcaption></figure>

**Access to model output or forecasts:** Real-time SEP predictions are publicly available via the HESPERIA project website: https://www.hesperia.astro.noa.gr/index.php/releases/real-time-prediction-tools/release. Projections are also available on the publicly-facing SEP scoreboards in addition to the iSWA interface hosted by CCMC.

**Model validation:** Analysis of historic data from 2009 to 2016 has shown the HESPERIA/RELEASE tools have a low FAR (≈30%) and a high POD (63%). (Malandraki et al., 2020). In Posner (2007), RELEASE was derived from a list of proton events, defined as >10 MeV exceeds 10 pfu, measured by GOES-8 from 1996–2002. The model was derived from COTSEP/EPHIN electron and proton measurements. At the time of the paper, the model had been running in real-time for 11 months in 2003. This time period was used for evaluation of the model. During this time, RELEASE issued nine series of SEP hazard warnings. ADWT, defined as the time of the issued forecast and the onset of the proton event (ons) or from proton intensity reaching the hazard level (haz), was also assessed.

Over the 11-month time period, 9 warnings were issued. 4 were correct warnings for SPEs that exceeded the hazard levels with an ADWT of 27, 18, 7, and 74 min (31.5 min average). Two false warnings were for SPE that did not cross the hazard thresholds with 30 and 7 min advance warning. The remaining 3 were false warnings which were due to increases in electron flux due to interplanetary shocks or occurred during the declining phase of an SPE. No mention was made about whether the model missed SPEs.

## 3.22. Sadykov et al. (2021) Model

**Model developers:** Viacheslav M. Sadykov, Alexander G. Kosovich, Irina N. Kitiashvili, Vincent Oria, Gelu M. Nita, Egor Illarionov, Patrick M. O'Keefe, Yucheng Jiang, Sheldon H. Fereira, Aatiya Ali (collaboration of

Georgia State University, New Jersey Institute of Technology, and NASA Ames Research Center).

**Model description:** The ML research model represents the artificial NN of a custom architecture designed for daily whole-Sun prediction of >10 MeV, >10 pfu SPEs. The model is trained and evaluated on the SC24 data and events. The NN architecture implemented in Sadykov et al. (2021) for the whole-Sun daily SPE prediction is schematically illustrated in Fig. 9. The key idea of this custom architecture is to pre-process the characteristics of sunspot groups and active regions (such as properties of SHARPS, Bobra et al., 2014) within so-called “AR Blocks” before combining them with other whole-Sun characteristics. The network parameters within the AR blocks (weights and biases of connections between neurons) are shared among all AR blocks, providing exactly the same preprocessing for each AR. Also, the outputs of the AR blocks are summed up before propagating to the main part of the network. This strategy allows one to (1) add the physics knowledge, that the ARs should be processed in the exact same way based on their characteristics and position on the solar disk, into the machine learning algorithm, and (2) significantly reduce the number of network parameters to tune while training the NN. The dark blue rectangles in Fig. 9 represent the fully-connected NN layers. The number of layers and neurons in each layer depends on the implementation and type of input. For example, the numbers of neurons in the subsequent layers were 30–15 - 8–4 - 2 for the AR blocks and 21–15 - 10–5 - 2 for the main network in Sadykov et al. (2021).

While the architecture demonstrated in Fig. 9 can be adapted to various settings of the whole-Sun SEP forecasts (particle energy and flux thresholds, prediction time windows, etc), Sadykov et al. (2021) considered it for daily prediction of the >10 MeV, >10 pfu proton events for the SC24. These particular energy and flux thresholds correspond to the least-strength S1 events on the scale of
<figure><img src="image_9.png" /><figcaption>Fig. 8. HESPERIA REleASE AWT for the SEP event on the 28th of March 2022.</figcaption></figure>

**Access to model output or forecasts:** Real-time SEP predictions are publicly available via the HESPERIA project website: https://www.hesperia.astro.noa.gr/index.php/releases/real-time-prediction-tools/release. Projections are also available on the publicly-facing SEP scoreboards in addition to the iSWA interface hosted by CCMC.

**Model validation:** Analysis of historic data from 2009 to 2016 has shown the HESPERIA/RELEASE tools have a low FAR (≈30%) and a high POD (63%). (Malandraki et al., 2020). In Posner (2007), RELEASE was derived from a list of proton events, defined as >10 MeV exceeds 10 pfu, measured by GOES-8 from 1996–2002. The model was derived from COTSEP/EPHIN electron and proton measurements. At the time of the paper, the model had been running in real-time for 11 months in 2003. This time period was used for evaluation of the model. During this time, RELEASE issued nine series of SEP hazard warnings. ADWT, defined as the time of the issued forecast and the onset of the proton event (ons) or from proton intensity reaching the hazard level (haz), was also assessed.

Over the 11-month time period, 9 warnings were issued. 4 were correct warnings for SPEs that exceeded the hazard levels with an ADWT of 27, 18, 7, and 74 min (31.5 min average). Two false warnings were for SPE that did not cross the hazard thresholds with 30 and 7 min advance warning. The remaining 3 were false warnings which were due to increases in electron flux due to interplanetary shocks or occurred during the declining phase of an SPE. No mention was made about whether the model missed SPEs.

## 3.22. Sadykov et al. (2021) Model

**Model developers:** Viacheslav M. Sadykov, Alexander G. Kosovich, Irina N. Kitiashvili, Vincent Oria, Gelu M. Nita, Egor Illarionov, Patrick M. O'Keefe, Yucheng Jiang, Sheldon H. Fereira, Aatiya Ali (collaboration of

Georgia State University, New Jersey Institute of Technology, and NASA Ames Research Center).

**Model description:** The ML research model represents the artificial NN of a custom architecture designed for daily whole-Sun prediction of >10 MeV, >10 pfu SPEs. The model is trained and evaluated on the SC24 data and events. The NN architecture implemented in Sadykov et al. (2021) for the whole-Sun daily SPE prediction is schematically illustrated in Fig. 9. The key idea of this custom architecture is to pre-process the characteristics of sunspot groups and active regions (such as properties of SHARPS, Bobra et al., 2014) within so-called “AR Blocks” before combining them with other whole-Sun characteristics. The network parameters within the AR blocks (weights and biases of connections between neurons) are shared among all AR blocks, providing exactly the same preprocessing for each AR. Also, the outputs of the AR blocks are summed up before propagating to the main part of the network. This strategy allows one to (1) add the physics knowledge, that the ARs should be processed in the exact same way based on their characteristics and position on the solar disk, into the machine learning algorithm, and (2) significantly reduce the number of network parameters to tune while training the NN. The dark blue rectangles in Fig. 9 represent the fully-connected NN layers. The number of layers and neurons in each layer depends on the implementation and type of input. For example, the numbers of neurons in the subsequent layers were 30–15 - 8–4 - 2 for the AR blocks and 21–15 - 10–5 - 2 for the main network in Sadykov et al. (2021).

While the architecture demonstrated in Fig. 9 can be adapted to various settings of the whole-Sun SEP forecasts (particle energy and flux thresholds, prediction time windows, etc), Sadykov et al. (2021) considered it for daily prediction of the >10 MeV, >10 pfu proton events for the SC24. These particular energy and flux thresholds correspond to the least-strength S1 events on the scale of
<figure><img src="image_10.png" /><figcaption>Fig. 9. A schematic representation of a neural network architecture for a daily whole-Sun prediction of SPEs based on properties of individual sunspot groups/ARs and whole-Sun integrated properties implemented in Sadykov et al. (2021) model. The neuron connections in AR Blocks share the weights and the biases.</figcaption></figure>

NOAA/SWPC<fnref n="12" />. In addition, this allows one to compare the prediction issued by machine learning algorithm with the operational daily probabilities of the S1 SPEs issued by NOAA/SWPC.

**Inputs:** The prediction is constructed based on the daily characteristics of the SXR and proton fluxes (e.g., the daily mean and median values), the daily median values of SHARP parameters in ARs (Bobra et al., 2014), and the daily counts of type-II, type-III, and type-IV radio bursts (Winter and Ledbetter, 2015). To account for the ARs near the western solar limb and at the far side of the Sun, the developers extrapolated the ARs for 10 days forward after the ARs pass the 68 deg west longitude assuming the Carrington rotation rate and unchanged SHARP parameters. The importance of the ARs near the western limb and related asymmetry of SEP events was demonstrated, for example, by Cliver et al. (2020) and was confirmed by the developers. The ratio of the number of samples in the train and test subsets was ~2/1, the class-imbalance ratio was kept ~1/34 in both subsets. Also, both subsets consisted of significantly-long temporal intervals to avoid temporal coherence problem (Ahmadzadeh et al., 2021).

**Outputs:** The model can provide both the binary and probabilistic (likelihood) predictions of the SPE to happen

during the next day (starting at 00:00 AM UT) based on the data available at or before 22:00 PM UT.

**Free parameters:** One can select the probability/ likelihood threshold to optimize the binary prediction with respect to a particular score (TSS, HSS, etc) or forecast setting (e.g. “all-clear”).

**Limitations and caveats:** The model developers noted that one of the problems in their approach is the relatively small data set used for model training and evaluation. The developers are currently expanding the “all-clear” prediction approach to the data and events of the SC22 and 23 based on the SXR and proton flux characteristics, as well as to the SC23 data and events based on the modified Zurich and magnetic classes of sunspot groups and ARs. Also, the non-operational (definitive) SHARP parameters (Hoeksema et al., 2014) were used. The performance of the model on near-real-time SHARP data has yet to be evaluated.

**Access to model output or forecasts:** For information on how to access the data set, the model, and related Jupyter Notebooks illustrating the data preparation and model training procedures, please see Sadykov et al. (2021).

**Model validation:** The model developers examined their prediction on the test data set in terms of several binary classification metrics, including traditionally used TSS and HSS, as well as binary cross-entropy, and the area under the ROC curve. The key results of their analysis are the following:

<footnote n="12">¹² https://www.swpc.noaa.gov/sites/default/files/images/NOAAscales.pdf.</footnote>
<figure><img src="image_10.png" /><figcaption>Fig. 9. A schematic representation of a neural network architecture for a daily whole-Sun prediction of SPEs based on properties of individual sunspot groups/ARs and whole-Sun integrated properties implemented in Sadykov et al. (2021) model. The neuron connections in AR Blocks share the weights and the biases.</figcaption></figure>

NOAA/SWPC<fnref n="12" />. In addition, this allows one to compare the prediction issued by machine learning algorithm with the operational daily probabilities of the S1 SPEs issued by NOAA/SWPC.

**Inputs:** The prediction is constructed based on the daily characteristics of the SXR and proton fluxes (e.g., the daily mean and median values), the daily median values of SHARP parameters in ARs (Bobra et al., 2014), and the daily counts of type-II, type-III, and type-IV radio bursts (Winter and Ledbetter, 2015). To account for the ARs near the western solar limb and at the far side of the Sun, the developers extrapolated the ARs for 10 days forward after the ARs pass the 68 deg west longitude assuming the Carrington rotation rate and unchanged SHARP parameters. The importance of the ARs near the western limb and related asymmetry of SEP events was demonstrated, for example, by Cliver et al. (2020) and was confirmed by the developers. The ratio of the number of samples in the train and test subsets was ~2/1, the class-imbalance ratio was kept ~1/34 in both subsets. Also, both subsets consisted of significantly-long temporal intervals to avoid temporal coherence problem (Ahmadzadeh et al., 2021).

**Outputs:** The model can provide both the binary and probabilistic (likelihood) predictions of the SPE to happen

during the next day (starting at 00:00 AM UT) based on the data available at or before 22:00 PM UT.

**Free parameters:** One can select the probability/ likelihood threshold to optimize the binary prediction with respect to a particular score (TSS, HSS, etc) or forecast setting (e.g. “all-clear”).

**Limitations and caveats:** The model developers noted that one of the problems in their approach is the relatively small data set used for model training and evaluation. The developers are currently expanding the “all-clear” prediction approach to the data and events of the SC22 and 23 based on the SXR and proton flux characteristics, as well as to the SC23 data and events based on the modified Zurich and magnetic classes of sunspot groups and ARs. Also, the non-operational (definitive) SHARP parameters (Hoeksema et al., 2014) were used. The performance of the model on near-real-time SHARP data has yet to be evaluated.

**Access to model output or forecasts:** For information on how to access the data set, the model, and related Jupyter Notebooks illustrating the data preparation and model training procedures, please see Sadykov et al. (2021).

**Model validation:** The model developers examined their prediction on the test data set in terms of several binary classification metrics, including traditionally used TSS and HSS, as well as binary cross-entropy, and the area under the ROC curve. The key results of their analysis are the following:

<footnote n="12">¹² https://www.swpc.noaa.gov/sites/default/files/images/NOAAscales.pdf.</footnote>