author: Naoto Nishizuka, Applied Electromagnetic Research Institute, National Institute of Information and Communications Technology, Tokyo 184-0015, Japan, firstname.lastname@example.com
Yûki Kubo, Applied Electromagnetic Research Institute, National Institute of Information and Communications Technology, Japan, firstname.lastname@example.com
Komei Sugiura, Department of Information and Computer Science, Keio University, Japan, firstname.lastname@example.com
Mitsue Den, Applied Electromagnetic Research Institute, National Institue of Information and Communications Technology, Japan, firstname.lastname@example.com
Mamoru Ishii, Applied Electromagnetic Research Institute, National Institute of Information and Communications Technology, Japan, email@example.com
title: Operational Solar Flare Prediction Model Using Deep Flare Net

# 1. Introduction

The mechanism of solar flares is a long-standing puzzle. Solar flares emit X-rays, highly energetic particles, and coronal mass ejections (CMEs) into the interplanetary space in the heliosphere, whereby these flares become one of the origins of space weather phenomena . The prediction of flares is essential for reducing the damage to technological infrastructures on Earth. Solar flares are triggered by a newly emerging magnetic flux or magnetohydrodynamic instability to release excess magnetic energy stored in the solar atmosphere . Such phenomena are monitored by the Solar Dynamic Observatory and the Geostationary Orbital Environment Satellite (GOES), and observation data are used for the prediction of flares.
Currently, flare prediction is tackled by the following four approaches: (i) empirical human forecasting , (ii) statistical prediction methods , (iii) machine learning methods , and (iv) numerical simulations based on physics equations . Some of the models have been made available for community use at the Community Coordinated Modeling Center (CCMC) of NASA . It is useful to show the robust performance of each model, and in benchmark workshops, prediction models were evaluated for comparison, where methods that included machine learning algorithms as part of their system were also discussed .
Recently, the application of supervised machine learning methods, especially deep neural networks (DNNs), to solar flare prediction has been a hot topic, and their successful application in research has been reported . However, there is insufficient discussion on how to develop the methods available to real-time operations in space weather forecasting offices, including the methods for validation and verification of the models. Currently, new physical and geometrical (topological) features are applied to flare prediction using machine learning , and it has been noted that training sets may be sensitive to which period in the solar cycle they are drawn from. .
It has been one year since we started operating our flare prediction model using DNNs, which we named Deep Flare Net . Here, we evaluate the prediction results during real-time operations at the NICT space weather forecasting office in Tokyo, Japan. In this paper, we introduce the operational version of DeFN in sections 2 and 3, and we show the prediction results in section 4. We propose the use of time-series cross-validation (CV) to evaluate operational models in section 5. We summarize our results and discuss the selection of a suitable evaluation method for models used in operational settings in section 6.

# 2. Flare Forecasting Tool in Real-Time Operation

## 2.1. Procedures of Operational DeFN

DeFN is designed to predict solar flares occurring in the following 24 h after observing magnetogram images, which are categorized into the two categories: ($\geq$M-class and $<$M-class) or ($\geq$C-class and $<$C-class). In the operational system of DeFN forecasts, observation images are automatically downloaded, active regions (ARs) are detected, and 79 physics-based features are extracted for each region. Each feature is standardized by the average value and standard deviation and is input into the DNN model, DeFN. The output is the flare occurrence probabilities for the two categories. Finally, the maximum class of flares occurring in the following 24 h is forecast by taking the maximum probability of the forecasts.
Operational DeFN was redesigned for automated real-time forecasting with operational redundancy. All the programs written in IDL and Python languages are driven by cron scripts at the prescribed forecast issuance time as scheduled. There are a few differences from the original DeFN used for research, as explained in the next subsection. A generalized flow chart of operational DeFN is shown in Figure 1.

## 2.2. NRT Observation Data

The first difference between development DeFN and operational DeFN is the use of near real-time (NRT) observation data. We use the observation data of line-of-sight magnetograms and vector magnetograms taken from Helioseismic and Magnetic Imager on board SDO, ultraviolet (UV) and extreme ultraviolet (EUV) images obtained from Atmospheric Imaging Assembly through 1600  Å and 131  Å filters; and the full-disk integrated X-ray emission over the range of 1–8  Å observed by GOES. For visualization, we also use white light images taken from HMI and EUV images obtained using AIA through 304  Å and 193  Å filters. The time cadence of the vector magnetograms is 12 min, that of the line-of-sight magnetograms is 45 s, those of the 1600  Å and 131  Å filters are both 12 s, and that of GOES is less than 1 min.
The data product of SDO is provided by the Joint Science Operations Center (JSOC) of Stanford University. The HMI NRT data are generally processed and available for transfer within 90 min after observations . This is why DeFN was designed to download the observation dataset 1 h earlier. If the observation data are unavailable because of processing or transfer delays, the target of downloading is moved back in time to 1 to 5 h earlier in the operational DeFN system. When no data can be found beyond 5 h earlier, it is considered that the data are missing. Here, the time of 5 h was determined by trial and error. Forecasting takes 20–40 min for each prediction; thus, it is reasonable to set the forecasting time to as often as once per hour. The 1 h cadence is comparable to that of the time evolution of the magnetic field configuration in active regions due to flux emergence or changes before and after a flare. However, DeFN started operating in the minimum phase of solar activity, so we started forecasting with a 6 h cadence instead of a 1 h cadence.
The NRT vector magnetograms taken by HMI/SDO are used for operational forecasts, whereas the calibrated HMI 'definitive' series of vector magnetograms are used for scientific research. The NRT vector magnetograms are accessed from the data series 'hmi.bharp_720s_nrt' with segmentations of 'field', 'inclination', 'azimuth', and 'ambig'. These segmentations indicate the components of field strength, inclination angle, azimuth angle, and the disambiguation of magnetic field in the photosphere, respectively. Additionally, the NRT line-of-sight magnetograms are downloaded from the data series 'hmi.M_720s_nrt', and the NRT white light images are from the 'hmi.Ic_noLimbDark_720s_nrt' (jsoc2) series. The NRT data of AIA 131  Å  193  Å  304  Å  and 1600  Å filters are retrieved from the 'aia.lev1_nrt2' (jsoc2) series.
Note that the HMI NRT vector magnetogram is not for the full disk, in contrast to the HMI definitive series data. HMI Active Region Patches (HARP) are automatically detected in the pipeline of HMI data processing , and the HMI NRT vector magnetogram is limited to the HARP areas plus a buffer, on which we overlaid our active region frames detected by DeFN and extracted 79 physics-based features . Furthermore, the correlation between the HMI NRT data and the definitive data has not been fully statistically revealed. A future task is to reveal how the difference between the HMI NRT and definitive series data affects the forecasting results. The same comments can be made for the AIA NRT and definitive series data.

## 2.3. Implementation of Operational DeFN

Operational DeFN runs autonomously every 6 h by default, forecasting at 03:00, 09:00, 15:00, and 21:00 UT. The forecasting time of 03:00 UT was set to be before the daily forecasting meeting of NICT at 14:30 JST. The weights of multi-layer perceptrons of DeFN were trained with the 2010-2014 observation datasets, and we selected representative hyperparameters by the observation datasets in 2015.
For the classification problem, parameters are optimized to minimize the cross entropy loss function. However, since the flare occurrence ratio is imbalanced, we adopted a loss function with normalizations of prior distributions. It is the sum of the weighted cross entropy.

Here, $p(y_{nk}^*)$ is the initial probability of correct labels $y_{nk}^*$, i.e., 1 or 0, whereas $p(y_{nk})$ is the estimated value of probability. The components of $y_{nk}^*$ are 1 or 0; thus, $p(y_{nk}^*)$=$y_{nk}^*$. $w_k$ is the weight of each class and is the inverse of the class occurrence ratio, i.e., \[1, 50\] for $\geq$M-class flares and \[1, 4\] for $\geq$C-class flares. Parameters are stochastically optimized by adaptive moment estimation with learning rate = 0.001, $\beta_1$ = 0.9, and $\beta_2$ = 0.999. The batch size was set to 150 .
Theoretically, the positive and negative events, i.e., whether $\geq$M-class flares occur or not, are predicted in the following manner. The following equation is commonly used in machine learning: $$\hat{y} = \mathop{\rm argmax}\limits_{k} p(y_k).$$ Here, $\hat{y}$ is the prediction result, and the threshold is usually fixed. For example, in the case of two-class classifications, the events with a probability greater than 50 % are output. When we use the model as a probabilistic prediction model, we also tried smaller threshold values for safety in operations, although there are no obvious theoretical meanings.
Note that the loss function weights cannot be selected arbitrarily. The positive to negative event ratios of $\geq$M-class and $<$M-class or $\geq$C-class and $<$C-class flares, which are called the occurrence frequency and the climatological base rate, are 1:50 and 1:4 during 2010-2015, respectively, in the standard. Only when the cross entropy is applied to the weight of the inverse ratio of positive to negative events does it become theoretically valid to output the prediction by equation (2). Therefore, we used the base rate as the weight of cross entropy.
The DNN model of the operational DeFN was developed as in Nishizuka et al. (2018). Because the full HMI and AIA datasets obtained from 2010 to 2015 were too large to save and analyze, the cadence was reduced to 1 h, although in general a larger amount of data is useful for better predictions. We divided the feature dataset into two for training and validation with a chronological split: the dataset obtained from 2010 to 2014 for training and the 2015 dataset for validation. The point of this paper is to contrast how well the DeFN model can predict solar flares in the real-time operations and in the research using time series CV methods (=shuffle and divide CV is insufficient). Then, we will discuss that the gap between the prediction accuracies in operations and in research using a time-series CV is small (see section 4.3).
The time-series CV is stricter than a K-fold CV on data split by active region. It might be true that a K-fold CV on data split by active region can also prevent data from a single active region being used in training and testing . However, a K-fold CV on data split by active region allows the training set to contain future samples from different active regions. This may affect the prediction results, when there is a long-term variation of solar activity. As well, the number of active regions which produced X-class and M-class flares is not so large that a K-fold CV on data split by active region may be biased and not equal.
Indeed, solar flare prediction in operation has been done in a very strict condition, where no future data is available. Our focus is not to deny a K-fold CV on data split by active region. Instead, our focus is to discuss more appropriate CVs in operational setting.
The model was evaluated with a skill score, the true skill statistic , which is a metric of the discrimination performance. Then, the model succeeded in predicting flares with TSS = 0.80 for $\geq$M-class and TSS = 0.63 for $\geq$C-class (Table 1). Note that the data for 2016–2018 were not used, because there were fewer flares in this period than in the period between 2010 and 2015.
Flare labels were attached to the 2010–2015 feature database for supervised learning. We collected on the disk all the flare samples that occurred from the flare event list. We visually checked the locations of the flares, compared them with NOAA numbers, and found the corresponding active regions in our database when there were two or more active regions. Then we attached flare labels to predict the maximum class of flares occurring in the following 24 h. If $\geq$M-class flares are observed within 24 h after observations, the data are attached with the label (0, 1)$_{\rm M}$; otherwise, they are attached with the label (1, 0)$_{\rm M}$. When two M-class flares occur in 24 h, the period with the label (0, 1)$_{\rm M}$ is extended. Similarly, the labels (0, 1)$_{\rm C}$ and (1, 0)$_{\rm C}$ are separately attached for the prediction model of C-class flares. The training was executed using these labels. On the other hand, in real-time operation, we do not know the true labels of flares, so we attached the NRT feature database with dummy labels (1, 0), which are not used in the predictions. It is possible to update the model by retraining it using the latest datasets if the prediction accuracy decreases. However, the pretrained operational model is currently fixed and has not been changed.

# 3. Operation Forecasts Using DeFN

## 3.1. Graphical Output

The graphical output is automatically generated and shown on a website (Figure 3). The website was designed to be easy for professional space weather forecasters who are often not scientists to understand. Prediction results for both the full-disk and region-by-region images are shown on the website, and the risk level is indicated by a mark, "Danger flares", "Warning", and "Quiet". Images are updated every 6 h as new data are downloaded. Details of the DeFN website are described below:

-   **Solar full-disk images and detected ARs:**
    Images obtained by multiwavelength observations, such as magnetograms and white light, 131  Å  193  Å  304  Å  and 1600  Å images taken by SDO, are shown along with ARs detected by DeFN, where the threshold is set to 140 G in the line-of-sight magnetograms taken by HMI/SDO .

-   **Probabilistic forecasts at each AR:**
    Probabilistic forecasts of flare occurrence at each AR are shown for $\geq$M-class and $<$M-class flares or $\geq$C-class and $<$C-class flares by bar graphs, in analogy with the probabilistic forecasts of precipitation. Note that this forecasted probability does not indicate the real observation frequency, because the prior distributions are normalized to the peak at 50 % by the weighted cross entropy, where the loss function weights are the inverse of the flare occurrence ratio . Thus, operational DeFN is optimized for forecasting with the default probability threshold of 50 %. That is, operational DeFN forecasts flares if the real occurrence probability, which is forecast by the non-weighted cross entropy loss function, is greater than the climatological event rate, and it does not forecast flares if the real occurrence probability is less than the climatological event rate . Therefore, the normalized forecasted probability of $\geq$M-class flares sometimes becomes larger than that of $\geq$C-class flares.

-   **Full-disk probability forecasts and alert marks:**
    The full-disk flare occurrence probability of $\geq$M-class flares, $P_{FD}$, is calculated using

    where $S$ is the set of ARs on the disk, $i$ is the number of ARs on the disk, element $AR_i$ is a member of set $S$, and $P_i$ is the probabilistic forecast at each $AR_i$ . The risk level is indicated by a mark based on $P_{FD}$ and is categorized into three categories: "Danger flares" ($P_{FD}$ $\geq$`<!-- -->`{=html}80 %), "Warning" ($P_{FD}$ $\geq$`<!-- -->`{=html}50 %), and "Quiet" ($P_{FD}$ $<$`<!-- -->`{=html}50 %). This is analogous to weather forecasting, e.g., sunny, cloudy, and rainy.

-   **List of comments and remarks:**
    Forecasted probabilities (percentages), comments, and remarks are summarized in a list.

## 3.2. Details of Operations and Redundancy

Operational DeFN has operated stable forecasts since January 2019. In this subsection, we explain the redundancy and operational details of operational DeFN.

-   **Forecast outages:** A forecast is the category with the maximum probability of a flare in each of the categories in the following 24 h after the forecast issue time. A forecast is normally issued every 6 h. If problems occur when downloading or processing data, the forecast is skipped and handled as a forecast failure.

-   **Data outages of SDO/HMI, AIA:** There are delays in the HMI/SDO data processing when no applicable NRT data are available for a forecast. In this case, the NRT data to download is moved back in time to 1 to 5 h earlier. In such as case, the forecasting target will change from 24 h to 25-29 h, though the operational DeFN is not retrained. If no data can be found beyond 5 h earlier, the "no data" value is assigned and the forecast is skipped.

-   **No sunspots or ARs with strong magnetic field on disk:** If there are no sunspots or ARs detected with the threshold of 140 G on the disk image of the line-of-sight magnetogram, feature extraction is skipped, a forecast of "no flare" with a probability forecast of 1 % is issued, and the "no sunspot" value is assigned.

-   **Forecasts at ARs near/across limb:** DeFN is currently not applicable to limb events. If an AR is detected across a limb, it is ignored in forecast targets.

-   **Flares not assigned to an active region:** Detected active regions by operational DeFN are not completely the same as active regions registered by NOAA. There are cases where flares occur in decaying or emerging active regions which are not detected by DeFN with the threshold of 140 G. This occurs most often for C-class and lower intensity flares, for example, C2.0 flare in NOAA 12741 on 2019 May 15. Such a flare is missed in real-time forecasts but included in evaluations.

-   **Retraining:** DeFN can be retrained on demand, and a newly trained model can be used for forecasting. Currently, the pretrained model is fixed and has not been changed so far.

-   **Alternative data input after SDO era (option):** Since DeFN is designed to detect ARs and extract features by itself, it can be revised and trained to include other space- and ground-based observation data in DeFN, even when SDO data are no longer available.

# 4. Forecast Results and Evaluation

## 4.1. Operational Benchmark

The purpose of machine-learning techniques is to maximize the performance for unseen data. This is called generalization performance. Because it is hard to measure generalization performance, it is usually approximated by test-set performance, where there is no overlap between the training and test sets.
On the other hand, as the community continues to use a fixed test set, on the surface the performance of newly proposed models will seem to improve year by year. In reality, generalization performance is not constantly improving, but there will be more models that are effective only for the test set. This is partially because that models with lower performance than state-of-the-art are not reported. In other words, there are more and more models that are not always valid for unseen datasets. It is essentially indistinguishable whether the improvement is due to an improvement in generalization performance or because it is a method that is effective only for the test set.
The above facts are well-known in the machine learning community, and the evaluation conditions are mainly divided into two, basic and strict. Under the strict evaluation conditions, only an independent evaluation body evaluates each model using the test set only once. The test set is not published to the model builders . The solar flare prediction is originally a prediction of future solar activity using a present observation dataset, and the data available to researchers are only the past data. This fact is consistent with the strict evaluation condition in machine-learning community.
In this section, we evaluate our operational forecasting results. We call this the "operational benchmark" in this paper. In the machine learning community, a benchmark using a fixed test set is used only for basic benchmark tests. The basic approach is simple but is known to be insufficient. This is because no one can guarantee that the test set is used only once. In strict machine learning benchmarks, evaluation with a completely unseen test set is required. Only organizer can see the "completely unseen test set", which cannot be seen by each researcher. This is because, if researchers use the test set many times, they implicitly tend to select models effective only for the fixed test set.
We think that the evaluation methods of operational solar flare prediction models are not limited to evaluations using a fixed test set. However, this paper does not deny the performance evaluation using a fixed test set. The purpose of this paper is to show that the operational evaluation is important. From a fairness perspective, the strict benchmarking approach takes precedence over the basic approach. Our operational evaluation is based on the strict benchmarking approach. We did not retrain our model after the deployment of our system.

## 4.2. Forecast Results and Evaluation

We evaluated the forecast results from January 2019 to June 2020, when we operated operational DeFN in real time. During this period, 24 C-class flares and one M-class flare were observed. The M-class flare was observed on 6 May 2019 as M1.0, which was originally reported as C9.9 and corrected to M1.0 later. The forecast results are shown in Table 2. Each contingency table shows the prediction results for $\geq$M-class and $\geq$C-class flares. operational DeFN was originally trained with the probability threshold of 50 % to decide the classification, but in operations, users can change it according to their purposes. In Table 2, we show three cases for $\geq$M-class and $\geq$C-class predictions using different probability thresholds, such as 50 %, 45 %, and 40 % for reference.
Each skill score can be computed from the items shown in contingency tables, and not vice versa. This is a well-known fact. No matter how many skill scores you show, you will not have more information than one contingency table. The relative operating characteristic (ROC) curve and the reliability diagram, which are shown in Leka et al. (2019), can also be reproduced from the contingency table if it is related to the deterministic forecast (forecast of this paper). The ROC curve is a curve or straight line made by plots on a probability of false detection (POFD) - probability of detection (POD) plane. The ROC curve for a deterministic forecast is made by connecting three points (0,0), (POFD, POD) for a deterministic forecast, and (1,1) . For reference, we introduce skill scores used in Leka et al. (2019), such as the accuracy, Hanssen & Kuiper skill score/Pierce skill score/True skill statistic (TSS/PSS), Appleman skill score (ApSS), equitable threat score (ETS), Brier skill score, mean-square-error skill score (MSESS), Gini coefficient, and frequency bias (FB).
According to Table 2, the flare occurrence was very rare and imbalanced in the solar minimum phase. Most of the forecasts are true negative. When we decrease the probability threshold, the number of forecast events increases. We evaluated our results with the four verification metrics in Table 3: accuracy, TSS, false alarm ratio (FAR), and Heidke skill score (HSS) . They show that operational DeFN optimized for $\geq$C-class flare prediction achieved accuracy of 0.99 and TSS of 0.70 with the probability threshold of 50 %, whereas they were 0.98 and 0.83 with the probability threshold of 40 %. DeFN optimized for $\geq$M-class flare prediction achieved accuracy of 0.99 but TSS was only 0.24 because only a single M1.0 flare occurred. Operational DeFN did not predict this flare because it was at the boundary of the two categories of $\geq$M-class and $<$M-class flares. This happens a lot in real operations, and this is a weakness of binary classification systems used in operational settings.
The trends of the contingency tables are similar to those evaluated in the model development phase. (Table 2). However, there are two differences. First, the data used were the NRT data, whereas the definitive series was used for development. However, in this case, there was negligible difference between them. Second, the evaluation methods are different. The operational DeFN was evaluated on the actual data from 2019 to 2020, whereas the development model was validated with the 2010-2014 dataset and tested with the 2015 dataset. It appears that the chronological split provides more suitable evaluation results for operations than the common methods, namely, shuffle and split CV and K-fold CV.

## 4.3. Time-series CV

Here we propose the use of time-series CV for evaluations of operational forecasting models. In previous papers on flare predictions, we used hold-out CV, where a subset of the data split chronologically was reserved for validation and testing, rather than the naïve K-fold CV. This is because it is necessary to be careful when splitting the time-series data to prevent data leakage . To accurately evaluate prediction models in an operational setting, we must not use all the data about events that occur chronologically after the events used for training.
The time-series CV is illustrated in Figure 4. In this procedure, there are a series of testing datasets, each consisting of a set of observations and used for prediction error. The corresponding training dataset consists of observations that occurred prior to the observations that formed the testing dataset and is used for parameter tuning. Thus, model testing is not done on data that may have pre-dated the training set. Furthermore, the training dataset is divided into training and validation datasets. The model prediction accuracy is calculated by averaging over the testing datasets. This procedure is called rolling forecasting origin-based CV . In this paper, we call it time-series CV, and it provides an almost unbiased estimate of the true error .
Note that the time-series CV has the following advantages: (i) The time-series CV is the standard validation scheme in time-series prediction. (ii) A single chronological split does not always reflect low generalization error . In other words, the trained model is not guaranteed to work for unseen test set. To avoid this, the time-series CV applies multiple chronological splits. The ability to predict new examples correctly that differ from those used for training is known as generalization performance . Therefore, the time-series CV is more generic and appropriate.
The evaluation results obtained by time-series CV using the 2010–2017 datasets are summarized in Table 4. The datasets were chronologically split to form the training, validation, and testing datasets. TSS is largest with the 2010–2014 datasets for training, the 2015 datasets for validation, and the 2016 datasets for testing. This is probably because it is not possible to obtain a reliable forecast based on a small training dataset obtained from 2010 to 2012. By averaging over the five testing datasets, we found that TSS is 0.70 for $\geq$M-class flares and 0.59 for $\geq$C-class flares. This procedure will be more suitable for an observation dataset with a longer time period.

# 5. Summary and Discussion

We developed an operational flare prediction model using DNNs, which was based on a research version of the DeFN model, for operational forecasts. It can provide probabilistic forecasts of flares in two categories occurring in the next 24 h from observations: $\geq$M-class and $<$M-class flares or $\geq$C-class and $<$C-class flares. DeFN has been continuously used for operational forecasting since January 2019, and we evaluated its performance using the forecast and actual flare occurrences between January 2019 and June 2020. We found that operational DeFN achieved an accuracy of 0.99 and TSS of 0.70 for $\geq$C-class flare predictions, whereas the accuracy was 0.99 but TSS was only 0.24 for $\geq$M-class flare prediction using a probability threshold of 50 %. using a probability threshold of 40 %, the accuracy was 0.98 and TSS was 0.83 for $\geq$C-class flares, whereas they were 0.98 and 0.48 for $\geq$M-class flares.
Operational DeFN has the advantages of a large TSS, good discrimination performance, and the low probability of missed detection of observed flares. This is why it is useful for operations that require that no flares are missed, such as human activities in space and critical operations of satellites. On the other hand, it tends to over-forecast and the false alarm ratio (FAR) increases. Because the number of true negatives is very large in an imbalanced problem such as solar flare prediction, TSS is less sensitive to false positives than to false negatives. Currently, the prior distributions of $\geq$M-class and $<$M-class flares are renormalized to increase TSS at threshold probability of 50 %, but this results in an increase in FAR.
When we compared the evaluation results, we observed no significant difference between the pretrained and operational results. This means that, at least during January 2019 – June 2020, the difference between NRT and definitive series science data did not greatly affect the forecasts. We found a TSS of 0.63 for the $\geq$C-class model evaluated using the pretrained model was maintained and even increased to 0.70 (0.83) for operational forecasts with the probability threshold of 50 (40) %. This suggests that the chronological split is more suitable for the training and validation of the operational model than shuffle and split CV.
Here, we discuss how to train and evaluate machine learning models for operational forecasting. For an exact comparison, it is desirable to use the same datasets among participants. If this is not possible, there are three points that require attention.

1.  Observation Database: The ratio of positive to negative events should not be artificially changed, and datasets should not be selected artificially. Data should be the climatological event rate and kept natural. This is because some metrics are affected by controlling the positive to negative event ratio of datasets, especially HSS, which will result in a difference from the operational evaluations. For operational evaluations, it is also desirable to include ARs near the limb, although they are excluded in most papers because the values of magnetograms are unreliable owing to the projection effect. Currently, in machine learning models, limb flares are not considered, but they also need to be considered in the near future, using GOES X-ray statistics as in human forecasting or magnetograms reproduced by STEREO EUV images .

2.  Datasets for Training and Testing: We recommend that a chronological split or time-series CV is used for training and evaluation of operational models. Although K-fold CV using random shuffling is common in solar flare predictions, it has a problem for a time-series dataset divided into two for training and testing when the time variation is very small, e.g., the time evolution of magnetic field. If the two neighboring datasets, which are very similar, are divided into both training and testing sets, the model becomes biased to overpredict flares. It might be true that a K-fold CV on data split by active region can also prevent data from a single active region being used in training and testing. However, a K-fold CV on data split by active region allows the training set to contain future samples from different active regions. Therefore, in the point of view of generalization performance, a time-series CV is stricter and more suitable for operational evaluation.

3.  Selection of Metrics: The ranking of models is easily affected by the selection of the metric. Depending on the purpose, users should select their preferred model by looking at the contingency tables and skill scores of each model. After understanding that each skill score can evaluate one aspect of performance, verification methods should be discussed in the space weather community .

In this paper, we showed contingency tables of our prediction results. No matter how many skill scores you show, you will not have more information than one contingency table. We evaluated our prediction results as a deterministic forecasting model. The ROC curve and the reliability diagram, which are shown in Barnes et al. (2016) and Leka et al. (2019), can also be reproduced from the contingency table if it is related to the deterministic forecast.
We demonstrated the performance of a machine learning model in an operational flare forecasting scenario. The same methods and discussion of prediction using machine learning algorithms can be applied to other forecasting models of space weather in the magnetosphere and ionosphere. Our future aim is to extend our model to predicting CMEs and social impacts on Earth by extending our database to include geoeffective phenomena and technological infrastructures.

# Declarations

# Availability of data and materials

The code is available at https://github.com/komeisugiura/defn18. In the README file, we explain the architecture and selected hyper parameters. The feature database of DeFN is available at the world data center of NICT (http://wdc.nict.go.jp/IONO/wdc/). The SDO data are available from the SDO data center (https://sdo.gsfc.nasa.gov/data/) and JSOC (https://jsoc.stanford.edu/). The GOES data are available at https://services.swpc.noaa.gov/json/goes/.

# Competing interests

The authors declare that they have no competing interests.

# Funding

This work was partially supported by JSPS KAKENHI Grant Number JP18H04451 and NEDO. A part of these research results was obtained within "Promotion of observation and analysis of radio wave propagation", commissioned research of the Ministry of Internal Affairs and Communications, Japan.

# Authors' contributions

N.N., Y.K. and K.S. developed the model. N.N. analyzed the data and wrote the manuscript. M.D. and M.I. participated in discussing the results.

# Preparing tables

| $\geq$M-class Flares |     | Observed Events |       |
|:--------------------:|:---:|:----------------|:------|
|         3-4          | Yes | No              |       |
|       Forecast       | Yes | 963             | 4382  |
|      2-4 Events      | No  | 54              | 25937 |

Contingency tables of DeFN using 2010-2015 datasets.

| $\geq$C-class Flares |     | Observed Events |       |
|:--------------------:|:---:|:----------------|:------|
|         3-4          | Yes | No              |       |
|       Forecast       | Yes | 4967            | 4420  |
|      2-4 Events      | No  | 1171            | 20778 |

Contingency tables of DeFN using 2010-2015 datasets.

| $\geq$M-class Flares |     | Observed Events |      |
|:--------------------:|:---:|:----------------|:-----|
|         3-4          | Yes | No              |      |
|       Forecast       | Yes | 1               | 28   |
|      2-4 Events      | No  | 3               | 2201 |

Contingency tables of DeFN forecasts in operation from January 2019 to June 2020. They show the forecast results for $\geq$M-class flares and for $\geq$C-class flares, with three different probability thresholds such as 50 %, 45 %, and 40 %.

| $\geq$C-class Flares |     | Observed Events |      |
|:--------------------:|:---:|:----------------|:-----|
|         3-4          | Yes | No              |      |
|       Forecast       | Yes | 27              | 18   |
|      2-4 Events      | No  | 11              | 2177 |

Contingency tables of DeFN forecasts in operation from January 2019 to June 2020. They show the forecast results for $\geq$M-class flares and for $\geq$C-class flares, with three different probability thresholds such as 50 %, 45 %, and 40 %.

| $\geq$M-class Flares |     | Observed Events |      |
|:--------------------:|:---:|:----------------|:-----|
|         3-4          | Yes | No              |      |
|       Forecast       | Yes | 1               | 31   |
|      2-4 Events      | No  | 3               | 2198 |

Contingency tables of DeFN forecasts in operation from January 2019 to June 2020. They show the forecast results for $\geq$M-class flares and for $\geq$C-class flares, with three different probability thresholds such as 50 %, 45 %, and 40 %.

| $\geq$C-class Flares |     | Observed Events |      |
|:--------------------:|:---:|:----------------|:-----|
|         3-4          | Yes | No              |      |
|       Forecast       | Yes | 30              | 27   |
|      2-4 Events      | No  | 8               | 2168 |

Contingency tables of DeFN forecasts in operation from January 2019 to June 2020. They show the forecast results for $\geq$M-class flares and for $\geq$C-class flares, with three different probability thresholds such as 50 %, 45 %, and 40 %.

| $\geq$M-class Flares |     | Observed Events |      |
|:--------------------:|:---:|:----------------|:-----|
|         3-4          | Yes | No              |      |
|       Forecast       | Yes | 2               | 34   |
|      2-4 Events      | No  | 2               | 2195 |

Contingency tables of DeFN forecasts in operation from January 2019 to June 2020. They show the forecast results for $\geq$M-class flares and for $\geq$C-class flares, with three different probability thresholds such as 50 %, 45 %, and 40 %.

| $\geq$C-class Flares |     | Observed Events |      |
|:--------------------:|:---:|:----------------|:-----|
|         3-4          | Yes | No              |      |
|       Forecast       | Yes | 32              | 34   |
|      2-4 Events      | No  | 6               | 2161 |

Contingency tables of DeFN forecasts in operation from January 2019 to June 2020. They show the forecast results for $\geq$M-class flares and for $\geq$C-class flares, with three different probability thresholds such as 50 %, 45 %, and 40 %.

| Probability threshold | Accuracy | TSS  | FAR  | HSS  |
|:----------------------|:---------|:-----|:-----|:-----|
| 50 %                  | 0.99     | 0.24 | 0.97 | 0.06 |
| 45 %                  | 0.98     | 0.24 | 0.97 | 0.05 |
| 40 %                  | 0.98     | 0.48 | 0.94 | 0.10 |

Evaluations of operational forecast results by DeFN from January 2019 to June 2020 with three verification metrics.

| Probability threshold | Accuracy | TSS  | FAR  | HSS  |
|:----------------------|:---------|:-----|:-----|:-----|
| 50 %                  | 0.99     | 0.70 | 0.40 | 0.64 |
| 45 %                  | 0.98     | 0.78 | 0.47 | 0.62 |
| 40 %                  | 0.98     | 0.83 | 0.52 | 0.61 |

Evaluations of operational forecast results by DeFN from January 2019 to June 2020 with three verification metrics.

| Datasets | TSS ($\geq$M-class flares) | TSS ($\geq$C-class flares) |
|:--:|:--:|:--:|
| Training (2010-2011), Validation (2012), Test (2013) | 0.49 | 0.53 |
| Training (2010-2012), Validation (2013), Test (2014) | 0.66 | 0.60 |
| Training (2010-2013), Validation (2014), Test (2015) | 0.77 | 0.66 |
| Training (2010-2014), Validation (2015), Test (2016) | 0.87 | 0.56 |
| Training (2010-2015), Validation (2016), Test (2017) | 0.72 | 0.61 |
| Average | 0.70 | 0.59 |

Evaluation of DeFN forecasts using the time-series CV.


abstract: Explosive growth in the amount of genomic data is matched by increasing power of consumer-grade computers. Even applications that require powerful servers can be quickly tested on desktop or laptop machines if we can generate representative samples from large data sets. I describe a fast and memory-efficient implementation of an on-line sampling method developed for tape drives 30 years ago. Focusing on genotype files, I test the performance of this technique on modern solid-state and spinning hard drives, and show that it performs well compared to a simple sampling scheme. I illustrate its utility by developing a method to quickly estimate genome-wide patterns of linkage disequilibrium (LD) decay with distance. I provide open-source software that samples loci from several variant format files, a separate program that performs LD decay estimates, and a C++ library that lets developers incorporate these methods into their own projects.
author: Anthony J. Greenberg [^1]
bibliography: tony.bib
title: Fast ordered sampling of DNA sequence variants

# Introduction

Growth in the amount of genomic data available is matched by increasing power and storage space of consumer-grade computers. Using such low-cost systems to perform genomis analyses can speed development cycles and empower users operating under economic constraints. The range of such analyses can be extended with light-weight software tools that carefully manage system resources.

Collections of single-nucleotide polymorphisms (SNPs) and copy number variants (CNVs) genotyped in groups of individuals are fundamental to numerous applications. These data sets are stored in a variety of formats and often contain millions of variants genotyped in thousands of individuals. It is often desirable to create random subsets of such large polymorphism tables. For example, relatively small samples can be used to quickly test software pipelines. In addition, when using genome-wide SNPs to predict phenotypes of individuals using genome selection methods, it is often important to learn the minimal marker set that achieves good accuracy . Finally, repeated creation of data subsets is a variant of the jack-knife procedure and can be used to construct empirical distributions of genome-wide statistics.

To be useful for a wide range of applications, any sampling scheme must meet several criteria. The subset generated must be in the same order as in the original data set. Variants must be sampled without replacement, each locus has to be picked with the same probability, and the size of the resulting data set must always reflect the value required by the user. The time required to sample variants must grow at most linearly with the number of polymorphisms in the sub-sample. Furthermore, because the original data set may be very large, the time required to pick loci must be as insensitive as possible to the overall size of the data set. In addition, since my aim is to empower researchers with limited access to powerful hardware, the implementation should minimize the use of system resources, particularly avoiding reading large files into memory. Surprisingly, after an extensive search I was unable to find existing software that performs according to these criteria. However, the general problem of ordered on-line sampling of records from files was solved 30 years ago . Unfortunately, this work is relatively unknown with limited application in computer system management and sampling of business data streams. No genetics papers appear to reference Vitter's articles.

I implemented a version of Vitter's algorithm that samples loci from a variety of variant file formats while minimizing system resource use. I examine the method's performance compared to a simple sampling scheme and provide an example application to estimate genome-wide patterns of linkage disequilibrium. I also provide a library that allows developers to incorporate these methods into their software. All source code, data, and analysis methods are openly available.

# Methods

## Sampling scheme

Vitter's main insight was to derive a scheme that samples the number of records to skip given how many remain to be picked and the number left in the file . There are additional speed-ups available if one is willing to store the index values in an array, but I opted to save memory instead and save the sampled records to the output file right away. Preliminary tests suggested that file I/O time dominates random number generation even on a machine with a solid state drive (SSD, results not shown), so the increase in sampling speed would not be noticeable.

Other than the above deviation, I implemented Vitter's method D as described in the Appendix A, algorithm A2 in . The implementation uses a hardware random number generator (RNG) (<https://software.intel.com/en-us/articles/the-drng-library-and-manual>). If not supported, the software substitutes the 64-bit Mersenne Twister seeded with the processor's time stamp counter. The decision is made automatically at run time and does not involve user input.

## Included software

This report describes three pieces of software: a C++ library `libsampFiles` and two stand-alone programs: `sampleSNPs` produces ordered samples from variant files and `sampleLD` uses locus samples to calculate distributions of linkage disequilibrium statistics. All software is released under the BSD three-part license. The whole set of programs can be trivially compiled using the included Makefile, with no external dependencies required. Compilation was tested on Mac OS with llvm/clang and on RedHat Linux using the GNU compiler collection.

#### C++ class library.

The `libsampFiles` library allows users to easily include in their own software support for sampling loci from most commonly used file formats (.tped and .bed from `plink` , VCF , and HapMap ), as well as a generic text and binary file. Reading and writing in these formats is supported, as well as limited manipulation (see the reference manual for details). Format conversion is not supported at present. Random number generators and population indexing facilities are also available. The library is constructed using hierarchical classes and is built with extensibility in mind. File manipulations are implemented to reduce random-access memory (RAM) use, without unduly reducing execution speed. The trade-offs were tested on a laptop with a solid state drive (SSD) and 16 gigabytes of RAM. Performance may differ on other system types.

In addition to the software, a directory with example SNP files is provided in the distribution for testing purposes. The project GitHub page (<https://github.com/tonymugen/sampleSNPs/>) provides a mechanism for users to report problems. Detailed library interface documentation is available at <https://tonymugen.github.io/sampleSNPs/>.

#### Sampling variants.

I used the `libsampFiles` library to write a stand-alone program, `sampleSNPs`, that subsamples variant files. All formats mentioned above are supported. The program runs via command line using standard Unix-style flags to pass execution parameters. The README file included with the project and available on the GitHub documentation page has detailed instructions. Sampled SNPs are saved into a file in the same format as the original. Auxiliary files, if present (e.g., .fam and .bim for .bed), are modified or copied as appropriate.

#### Linkage disequilibrium among sampled loci.

As an example of an application of locus sampling, I implemented a stand-alone program that estimates genome-wide LD decay with between-locus distance. A full accounting of this relationship would require the calculation of linkage disequilibrium statistics for all $N_p = n(n-1)/2$ pairs of loci, where $n$ is the number of genotyped variants. This task quickly becomes unmanageable as the number of genotypes in the data set grows. One solution, implemented in `plink` , is to calculate LD only among loci falling within a neighborhood window on a chromosome. A complementary approach: implemented here, is to sample $2N_s$ ($N_s$ is the desired number of sampled pairs) loci using Vitter's method and calculate LD between consecutive pairs. Justification for this approach is provided in Appendix A. Once a pair of loci is picked, `sampleLD` calculates two linkage disequilibrium statistics: $r^2$ and $D^{\prime}$ . Missing data are removed (only individuals successfully genotyped at both loci are considered). If there are not enough genotypes to produce a meaningful result, "-9" is reported. If a file with a population index is provided, the program will calculate LD statistics within each population and report them separately.

Unlike `sampleSNPs`, `sampleLD` currently only supports `plink` .bed files as input. The auxiliary .bim and .fam files are also required. A detailed description of input file requirements, command line flags, and output format are in the README file included with the project and on the documentation page.

## Test data

Execution timing was performed with SNP files extracted from the *Drosophila* genome nexus . I used the Zambia and France populations from that data set. LD measurements were performed on cultivated rice (*Oryza sativa*) genotypes . I extracted a random sample of 100 *indica* (IND) and 100 tropical *japonica* accessions, and filtered out loci with minor allele counts less than two. I estimated the smoothed relationships between LD and distance, with their confidence intervals, using the `ggplot2` R package .

# Results and Discussion

After an extensive search I was unable to find existing software that performs uniform sampling of loci from files without replacement while preserving their order. The widely-used command-line tool `plink` does have a function (accessible via the `--thin` flag) that extracts a random sample of SNPs while preserving order. However, the program simply examines each locus and includes it with the specified probability. Thus, the resulting sample varies in size (Supplemental Fig. S).

## Ordered sampling of loci

Given that no other software appears to be available, I set out to implement a light-weight solution that quickly generates ordered samples without replacement even from very large data sets. The simplest idea is to examine every variant record in turn and decide, given the current number of loci picked, the number remaining in the input file, and the total to be sampled, whether to pick the current one . The selected records are read into memory and saved to the output file. While this solution is obvious and easy to implement, it requires an examination and a (pseudo)random number sampling step for each line in the input file.

An alternative approach has been proposed by Vitter . The idea is to decide how many loci to skip, given the current number already picked and remaining to be examined. demostrated that this approach (Vitter's Method D) is faster than the simple line-wise decision-making outlined above (referred to as Method S). However, the tests were performed 30 years ago. The files were stored on tape, and random number generation was computationally expensive. Therefore, I implemented both Method S and Method D in C++, using comparable language facilities (see Methods and Supplemental files for details), and tested them in a number of scenarios to determine which scheme is preferable on modern-day computers.

Several variables can influence algorithm execution speed. Random (or pseudorandom) number generation is used extensively to generate samples from required distributions. However, code profiling (not shown) revealed that at least the hardware RNG I chose for this implementation (see Methods for details) is never the rate-limiting step, even when files are stored on a fast solid state drive. Rather, it is file input/output that takes most time during a given execution cycle. There are two parameters to consider when we investigate file read/write timing. One, storage can be either on a solid-state (SSD) or a spinning drive (HDD). The former is generally faster and allows for random file access. Second, the files can be either in a binary format (I use the popular and highly-compressed `plink` `.bed`), or in plain text. The important difference is that lines in files with text records are in general variable in length. Thus, if we want to skip several records we have no choice but to read each row in turn and discard unwanted loci. In contrast, binary formats use fixed-size fields, leading to uniform row sizes. It is then trivial to compute the number of bytes to skip without reading before arriving at the desired record.

Given that I am interested in creating a tool that can be used on personal workstations and laptops, and since solid state drives have become the standard choice, I focus on execution timing on a laptop (mid-2015 15-inch MacBook Pro) with an SSD. However, I also replicated the results on a 2014 Mac Mini with an HDD with essentially the same results (see Supplemental Figures and ). I first held the input file size constant and varied the number of loci sampled. As shown in Fig. , time taken by both Method D and Method S grows approximately linearly with the number of loci sampled. This is the case for both binary and text files. Method D (Vitter's skip-over algorithm) outperforms the simpler Method S several-fold when the number of records is much smaller than the total in a binary file. This is not surprising, given that in this case Method S examines and discards many more loci for each one picked. As expected, the difference largely disappears when we sample from a text file. This is because both methods have to at least read and discard file lines one at a time. Interestingly, I obtained similar results on an HDD (Supplementary Fig.  and ), even though a spinning drive should not allow the same level of random access as an SSD. It is notable that in every case working with a binary file is about an order of magnitude faster, even though I am sampling more loci from a bigger file (500,000 loci in the binary *vs* 100,000 in the text file). Finally, although the performance benefit of Method D is not always dramatic, it never underperforms Method S. The relatively small amount of extra time taken by Method S likely reflects the additional operations that are necessary to decide whether to include a given record.

Given that Vitter's method decides ahead of time how many loci to skip before sampling, I would expect that it should be relatively insensitive to the total size of the input data set. Indeed, this is the case for binary file sampling (Fig. , panels A and B). Increasing input file size 2.5-fold results in no measurable rise in execution time. Method S execution time, and that of both methods on a text file (Fig. , C and D), grows approximately linearly with input size. Again, Method D is always at least as fast as Method S.

Given that Method S never consistently outperforms Vitter's Method D, I included only the latter in my implementation. While I do include the facility to read various text variant file formats, it is clear that using the `.bed` binary files, ideally on an SSD, results in optimal performance.

## Linkage disequilibrium distributions

Estimating rates of LD decay with distance on a chromosome are necessary, for example, in genome-wide association studies where such rates determine peak resolution. Because calculating LD between all pairs of loci is infeasible and unnecessary, a typical approach is to estimate linkage statistics in sliding windows. This technique is employed in `plink`. I implemented an alternative approach, picking loci according to Vitter's algorithm (see Methods for details) and then calculating LD statistics between consecutive variants in the sample. To test my implementation, I used a data set of 638,699 SNPs from the rice high-density array . I first ran `plink` to calculate $r^2$ and $D^{\prime}$ between loci no more than 500 kb or 20 SNPs apart. This yields more than 12 million locus pairs, stored in a 1.2 gigabyte file. The relationships between linkage disequilibrium and distance are depicted in Fig. (A, B). As expected, precision of LD estimates between distant loci diminishes due to undersampling. I then analyzed the same data set using my approach, sampling 30,000 SNP pairs (the resulting file occupies a mere 1.4 megabyte). While the confidence intervals from these estimates are wider (Fig. A, B), the pattern of LD decay is the same as that captured by the considerably larger sample set produced by `plink`. Thus, my light-weight approach may be the best option when great precision is not required and computational resources are limited.

An extra feature of my `sampleLD` program, unavailable in `plink`, is the ability to make separate LD estimates for each population present in a set of individuals. I illustrate this possibility by estimating linkage disequilibrium in *indica* and tropical *japonica* rice varietal groups (Fig. C, D). It is well established that LD levels are lower in *indica*. My analyses recapitulate this pattern.

The software described in this report enables users to quickly generate subsets of large SNP or CNV data sets, opening the door to numerous applications that constrain resources available for genetic data manipulation. This work exemplifies the kinds of approaches needed to speed discovery cycles and empower researchers lacking access to expensive hardware.

# Web resources

Project name:

:   sampleSNPs

Project homepage:

:   <https://github.com/tonymugen/sampleSNPs/>

Project documentation:

:   <https://tonymugen.github.io/sampleSNPs/>

Operating systems:

:   Unix-like (Linux, BSD, Mac OS)

Programming language:

:   C++

Other requirements:

:   No dependencies other than the C++11 standard library

License:

:   BSD three-part

# Acknowledgements

The idea for this project was spurred by Kevin Lawler's blog post on Vitter's work (<https://getkerf.wordpress.com/2016/03/30/the-best-algorithm-no-one-knows-about/>). The data for implementation testing were obtained from the *Drosophila* Genome Nexus (<http://www.johnpool.net/genomes.html>) and the Rice Diversity website (<https://ricediversity.org/data/index.cfm>).

# Supplemental Data

#### Supplemental file 1 – Timing of sampling schemes

This is an archive of the directory that contains the R and C++ code, as well as data files, necessary to reproduce the algorithm timing analyses presented in this manuscript. Compilation and running instructions are included.

#### Supplemental file 2 – LD analyses

This is an archive of the directory that contains the R code and data files necessary to reproduce the linkage disequilibrium results presented in this report.

#### Supplemental file 3 – Software source code

This is an archive of the directory that contains the source code of software described in this paper. Compilation and testing instructions are included. An up-to-date version can be found on GitHub (<https://github.com/tonymugen/sampleSNPs/>).

# Appendix A: LD sample scheme derivation

For the sample to accurately reflect whole-genome values, the probability of picking each pair must be equal. Furthermore, pairs must be sampled without replacement. To derive such a scheme, I order the list of all possible locus pairs as they would appear in a pair-wise relationship matrix. Since LD measures are symmetric, we need only concern ourselves with the upper (or, equivalently, lower) triangle of this matrix. The first row of the upper triangle lists the pairings of the first locus on a chromosome with all $n - 1$ subsequent loci other than itself. Next row lists $n - 2$ relationships between the second variant and the rest, excluding itself and the first locus. The process continues until we reach the last locus, which has no additional pairs. Sampling from this list will yield the desired uniform representation of locus pairs. Each variant on the list of pairs is represented $n - i$ times in a row, where $i = 1, \dots, n-1$ is the index of the locus position. Thus, instead of going through the pairs list (which contains $N_p = n(n - 1)/2$ elements) we can use a two-step scheme. We start by picking the first locus in a pair by sampling variants with weights reflecting the length of their run in the pairs list. We would then randomly pick another variant from the remaining loci. Finally, we go back to the first SNP or CNV in the pair and use it to sample the jump length to the next locus according to a weighted algorithm and repeat the process until we have the desired number of pairs. The initial sampling weight for locus $i$ under this scheme is

$$w_i = \dfrac{p_i}{\sum_i p_i},$$ where $p_i$ is the probability of sampling locus $i$. Since, as mentioned above, each variant is represented $n - i$ times on the list,

$$p_i = \dfrac{n - i}{N_p} = \dfrac{n - i}{n(n - 1)/2}$$ Since $p_i$ are probabilities, $\sum_i p_i = 1$. This leads to the expression for $w_i$: $$\begin{aligned}
w_i &=\quad \dfrac{p_i}{\sum_i p_i}\\
 &=\quad p_i\\
 &=\quad \dfrac{2(n - i)}{n(n - 1)}\\
 &=\quad \dfrac{2}{n-1} - \dfrac{2i}{n(n-1)}\\
 &\approx\quad \dfrac{2}{n} - \dfrac{2i}{n^2} \quad\text{when $n$ is large}
\end{aligned}$$ Thus, the deviation from an equal-weight random sampling (with all $w_i = \tfrac{1}{n}$) depends solely on the value approximately $\tfrac{2i}{n^2}$, which is tiny for large data sets ($n \ge 100,000$) we typically encounter.

According to the scheme presented above, once we have the first locus in a pair, we would then sample randomly from the loci further down on the chromosome to obtain the second variant for LD calculations. The next round would then require us to go back in the file to the first locus in the pair and continue with our scheme. This step is potentially computationally expensive, especially for text files with variable-width lines. To eliminate this complication, I further simplify the algorithm by instead using Vitter's method to sample $2N_s$ ($N_s$ is the desired number of sampled pairs) loci. I then calculate LD between consecutive pairs of variants. A slight correction is needed only when more than one chromosome is present in the data set. In such cases, locus pairs that are located on different chromosomes are discarded and the additional pairs are sampled to restore the total to the required value. The resulting scheme approximates true uniform sampling very well when data sets are large and sample sizes are relatively small (preliminary tests suggested that sample sizes as large as 1/3 the total number of SNPs still yield reasonable results).

# Figures

[^1]: Bayesic Research, Ithaca, NY, USA; email@example.com