DOI: 10.5281/zenodo.4718560
title: [Data] Prediction of soft proton intensities in the near-Earth space using machine learning

description: <strong>1 Introduction</strong> The dataset consists of four files: <pre><code>File Name Type RAPID_OMNI_ML_023_raw.h5 dataset RAPID_OMNI_ML_023_traincut.h5 dataset RAPID_OMNI_ML_023_testcut.h5 dataset RAPID_OMNI_ML_023_robusttranscut.pkl pre-fit scaler</code></pre> <em><strong>1.1 Variables</strong></em> The dataset files contain the following variables. <pre><code>Variable Name Type Unit Description p1, p2, p3, float 1/(s⋅cm²⋅sr⋅keV) proton intensities p4, p5, p6, p7 x, y, z float RE position in GSE coordinates rdist float RE radial distance from the Earth AE_index float nT Auroral Electroject (AE) index SYM-H_index float nT symmetric disturbance field in horiz. direction F107 float sfu the solar radio flux at 10.7 cm BimfxGSE, float nT x, y and z components of the Interplanetary BimfyGSE, Magnetic Field in GSE coordinates BimfzGSE VxSW_GSE, float km/s x, y and z components of the solar wind speed VySW_GSE, VzSW_GSE NpSW float n/cc solar wind density Temp float K solar wind temperature Pdyn float nPa solar wind dynamic pressure DateTime datetime timestamp </code></pre> <strong><em>1.2 Data Split Ranges</em></strong> The datasets contain data from the following time ranges. <pre><code>Dataset Start End Count RAPID_OMNI_ML_023_raw.h5 2001-01-09 15:21:00 2018-02-19 09:57:00 6,051,937 RAPID_OMNI_ML_023_traincut.h5 2001-01-09 15:21:00 2014-07-24 22:44:00 4,524,200 RAPID_OMNI_ML_023_testcut.h5 2014-07-24 22:45:00 2018-02-19 09:57:00 1,173,865</code></pre> <strong>2 Raw Data Preparation</strong> <em><strong>2.1 Data Source</strong></em> <em>OMNIWeb (NASA)</em> From NASA/GSFC's OMNI data set through OMNIWeb, we extracted the following variables from 2001 to 2019: <pre><code>Variable Original Name in OMNIWeb Resolution AE_index AE Index, nT 1-min SYM-H_index SYM/H, nT 1-min F107 Solar index F10.7 1-hour * BimfxGSE Bx, GSE/GSM, nT 1-min BimfyGSE By, GSE, nT 1-min BimfzGSE Bz, GSE, nT 1-min VxSW_GSE Vx Velocity, GSE, km/s 1-min VySW_GSE Vy Velocity, GSE, km/s 1-min VzSW_GSE Vz Velocity, GSE, km/s 1-min NpSW Proton Density, n/cc 1-min Temp Proton Temperature, K 1-min </code></pre> <code>*</code> Solar index F10.7 is not available at higher resolution. <em>Cluster Science Archive (ESA)</em> Through the Cluster Archive Inter-Operability Subsystem, we have access to the proton intensities in 7 energy channels measured by RAPID instrument onboard Cluster satellite. We got the following variables between 2001 and 2009 from the CDF files: <pre><code>Variables Dataset ID in CAIO Variable in CDF Files Resolution p1, p2, p3, p4, C4_CP_RAP_HSPCT Proton_Dif_flux__C4_CP_RAP_HSPCT 4127-ms p5, p6, p7 x_km, y_km, z_km* C4_CP_AUX_POSGSE_1M sc_r_xyz_gse__C4_CP_AUX_POSGSE_1M 1-min</code></pre> <code>*</code> Positions in km are not included in the final dataset. <em><strong>2.2 Custom Features</strong></em> <em>OMNIWeb (NASA)</em> <pre><code>Variable Source Pdyn NpSW * (VxSW_GSE^2 + VySW_GSE^2 + VzSW_GSE^2) * 1.67e-6</code></pre> <em>Cluster Science Archive (ESA)</em> <pre><code>Variable Source x x_km / 6371.1 y y_km / 6371.1 z z_km / 6371.1 rdist sqrt(x^2 + y^2 + z^2)</code></pre> <em><strong>2.3 Sampling and Interpolation</strong></em> - We use the value of <code>F107</code> at a 1-hour resolution to represent all values in each 1-hour bin. - For successful integration, we sampled the proton intensities to a resolution of 1 minute. More specifically, we calculate the averaged proton intensities for seven channels in each minute and use them to represent the values at first second in each minute, e.g., values at 2001/1/9 15:22:00 are calculated with the data from 15:22:00 - 15:22:59. <em><strong>2.4 Integration</strong></em> As now data from different sources can be aligned with <code>Datetime</code> at a resolution of 1 minute, we can merge them. <strong><em>2.5 Cleaning</em></strong> At last, we dropped the rows with outliers (fill values) in any OMNI variable. Please refer to the in the description from OMNIWeb for more information about the fill values. The raw data generated is available in the package with the name <code>RAPID_OMNI_ML_023_raw.h5</code>. <strong>3 Experiment-specific Pre-processing</strong> Besides, we took the steps below to the dataset for our experiments. <em><strong>3.1 Splitting</strong></em> The dataset is split into a training set and a test set with a ratio of 8:2. <em><strong>3.2 Filtering</strong></em> - We filtered out the rows with <code>rdist</code> less than or equal to 6. - We also use NaNs to substitute the proton intensities less than or equal to the threshold, which is 5, 1, 0.5, 0.1, 0.05, 0.005 or 0.001 respectively for 7 channels. <em><strong>3.3 Transform</strong></em> - We did not perform any transform or scaling directly on the data in pre-processing. Instead, a Robust Scaler fit with the training data was dumped as a file and used in the experiment. The pre-processed data and scaler are available under names <code>RAPID_OMNI_ML_023_traincut.h5</code>, <code>RAPID_OMNI_ML_023_testcut.h5</code> and <code>RAPID_OMNI_ML_023_robusttranscut.pkl</code>.