dataesr/scientific-paragraphs-categorization huggingface dataset:
Thus it is important to understand how a supervised method may depend on the training dataset if it is to be applied to a dataset of a slightly different nature. It is common practice that data from climate simulations are first produced and stored and then analyzed. However, the climate modeling community is also moving in the direction of "on-the-fly" (also called in situ) data analysis in order to reduce the volume of data to be stored and the environmental impacts of such storage. This paradigm change implies the development of more efficient analysis methods. Both physical algorithms (i. e. , trackers) and DL models are legitimate approaches to study TCs, but it is useful to understand if one generalizes better than the other. However, it is important to understand that both approaches do not necessarily achieve the same thing.
Indeed, trackers search for the trajectory of a cyclone by detecting its different positions in time, whereas the DL models listed in Table  1 , derived from computer vision, detect cyclones on an image frozen in time. In this context and for the above-mentioned reasons, we have developed in this study a detailed procedure for building training datasets and testing the performance of the TC detection algorithm to some of its parameters. We chose to work with two reanalyses (ERA5 and MERRA-2) and view this as a necessary first step before being able to apply our methodology to climate simulations. In Sect. 1, we present the data used to generate the images to be classified. Then in Sect. 2, we explain the architecture of the classification model, its training, the evaluation method to assess its performances, and the processes for generating the images to be classified.
TC dataset Several datasets of TCs exist: we can flag here Ex-tremeWeather  (Racah et al. , 2017) , ClimateNet  (Prabhat et al. , 2020) , and the International Best Track Archive for Climate Stewardship (IBTrACS,  Knapp et al. , 2010, and references therein). In this study we use the North Atlantic National Hurricane Centre (NHC) "best track" hurricane   and Franklin, 2013)  because it is known as a high-quality dataset for the North Atlantic basin. Quality and quantity of the training dataset are essential for the accuracy and performance of ML models. In particular it is important for the dataset to be comprehensive (i. e. , there is no missed TC) and homogeneous (i. e. , the criteria for deciding if a feature qualifies as a TC are used consistently in space and time). The HURDAT2 dataset is reputed to be comprehensive for the period after 1970  (Landsea et al.
Images In computer vision, the term image refers to a stack of matrices (also called a 3D tensor), with each matrix representing an information channel. For example, RGB images are formed of a stack of matrices of numerical values coding the red (R), green (G), and blue (B) color intensities of each pixel of a photograph. Our use of the term image is a generalization of the concept of RGB images. In the rest of our study, an image refers to a stack of gridded data extracted from a different variable of ERA5 or MERRA-2 in a given geographical area. Unlike for an RGB image, the channels cannot be combined; we thus graphically represent each channel separately.
In this study, we use the conventional approach of binary classification by considering one output neuron activated by a sigmoid function. So a probability value that tends to 0 classifies an image as background, while a value that tends to 1 classifies an image as cyclone. By construction, the size and the number of channels of the input images in a CNN are fixed. Using different image sizes and/or numbers of channels would require modifying the network architecture and retraining it. Indeed, the properties of the dense layers of the network depend on the image shape (i. e. , the number of neurons). Thus, image classification using a CNN implies the production of training and testing datasets of a given shape, irrespectively of the atmospheric reanalyses, ERA5 or MERRA-2, being considered.
Principles The training of a CNN classifier is based on the optimization of its parameters using gradient descent and backpropagation techniques. Roughly speaking, the training process presents a batch of images as an input to the CNN. The training process modifies the parameters of the CNN in order to improve the classification of the batch, according to a chosen loss function. For a binary classifier, this process implies the presentation of images containing a TC but also images not containing a TC, called background images. We now explain the data engineering involved in selecting both TC-containing and background images using the HURDAT2 dataset of cyclone tracks and the ERA5 and MERRA-2 reanalyses. https://doi. org/10. 5194/gmd-15-7051-2022 Geosci. Model Dev. , 15, 7051-7073, 2022.
TC-containing image generator The HURDAT2 dataset provides locations and dates of TCs as part of the cyclone metadata. We create images centered on the cyclone positions in the reanalysis for the dates indicated in HURDAT2. The different channels of the images consist of the selected variables from the reanalyses as discussed above. We consider all cyclones with HU and TS status (see Table  A1 ) that are located over the ocean, islands, and coasts, over the period 1980-2019. Most TCs are found during the Atlantic hurricane season from May to December, but we also consider a few events identified by HURDAT2 as TCs outside these months. Figure  1  shows the spatial and temporal distributions of the TC-containing images.
Background image metadata generator Extracting background images requires some thought because the performance of the CNN depends on these images and whether these images sample the diversity of TC-free situations. The idea here is to reuse the HURDAT2 database so that, for each location and date with a TC, we choose two dates in the past where no TC is present. We also check that the date was not already selected as the TC-free situation for another TC-containing image, so that all background images are distinct to each other. Once the dates are selected, we can extract the corresponding images. Figure  2   our background images do not include a TC by construction; the meteorological, geographical, and temporal contexts of the background images are close to those of the TCcontaining images generated on the basis of the HUR-DAT2 data -in this way, we hope to better train the model at the classification decision boundaries; the ratio of background over TC-containing images is constant by construction (with one-third TC-containing images and two-thirds background images); the background images cannot be within 48 h from a cyclone image and 12 h from another background image, considering the geographical domain.
Image interpolation Previously we have detailed the automatic production chain of constant-shape images to satisfy the constraints of the CNN. However, as mentioned above, the ERA5 and MERRA-2 reanalyses do not have the same spatial resolution (0. 25 versus 0. 5 • ). In order for the images to represent a constant domain size and thus include cyclone of the same size as a fraction of the image domain size, we extract native images of 16×16 pixels for MERRA-2 and 32×32 pixels for ERA5 as described in Table  5. We then symmetrize the MERRA-2 native image set at a resolution of 0. 5 • × 0. 5 • with a bilinear interpolation to obtain the MERRA-2 16px@0. 5 image set. To resolve the difference in resolution and to study the sensitivity of the CNN to the different datasets, we further transform by bilinear interpolation one of the image sets to the properties of the other set (image resolution and ERA5 native 32 × 32 0.
The obtained values, described in Table  6 , are used for all experiments to avoid attributing the variability in the studied metrics to hyperparameter changes. These metrics and the methods for evaluating them are the subject of the next section. We did not try to optimize the architecture of the CNN proposed by  Liu et al. (2016)  because we believe they have already optimized it and the modifications we have made do not require any further optimization, since the performance of our CNN is very close to that of  Liu et al. (2016). Finally, as explained above, we prefer to focus on the performance and the sensitivity of the CNN to the learning dataset instead of obtaining better performances.
Evaluation of metrics In our study, we used three classical metrics to measure the performance of our binary classification model: accuracy, the https://doi. org/10. 5194/gmd-15-7051-2022 Geosci. Model Dev. , 15, 7051-7073, 2022  The AUC measures the power of a model to discriminate between the two classes for a variety of decision threshold values. The AUC is, as the acronym indicates, the area under the ROC curve. The latter is depicted by plotting the recall of a model against the false positive rate of the same model. The recall measures the ability of a model to identify all occurrences of a class. The recall and false positive rate values are calculated according to the ground truth and the classifier responses for a given test dataset and for all possible decision thresholds (or a set of discrete values).
To that end, we applied an iterative cross-validation method, which consists in repeating a cross-validation method 20 times. We chose the k-fold method  (Bishop, 2006) , with k equal to 10, as the cross-validation method. We obtained a mean of the metrics for each k-fold iteration. By applying the central limit theorem on this set of metric means, we could compute the expected value and the uncertainty in the metrics. In order to avoid any bias, we took care to check if the central limit theorem can be applied by testing the normality of the distribution of the metric means using the Shapiro-Wilk statistical test (brief non-mathematical presentation given in Appendix B1) for an alpha level of 1 %. Moreover, images coming from a time series of tracks from the same cyclone may be found in both the training and test datasets, which would induce some dependance between the training and test datasets due to the autocorrelation within individual cyclone tracks.
By default, ML libraries set the threshold to 0. 5; however, in our case, the optimal threshold, indicated in the title of each panel of Fig. 6 , is lower than 0. 5, and for some combination of training and testing datasets, it is even much lower (e. g. , ERA5/MERRA-2 combination in 16px@0. 5). This reflects the fact that (i) the image sets are not balanced and (ii) the number of false negatives (orange color on the left side) is larger than the number of false positives (blue color on the right side) for this particular partitioning. Our set of experiments shows that the choice of the threshold value depends on the partitioning, the source of the data, and the relative importance given to false negatives and false positives. While accuracy is a less interesting metric than AUC and AUPRC, we decided to keep it, as a matter of information, and set its threshold to 0.
Additionally, the model of  Shakya et al. (2020)  is trained and tested on observational data that are quite different from multidimensional meteorological reanalysis data. The values of the metrics are very high, but it does not mean that a model is useful. Indeed, the usefulness of a model is measured by the difference between its performance and that of models based on simple rules or a domain-specific baseline. For instance, we implement the following simple models (from the software library scikit-learn): "most frequent", which always predicts the most frequent class observed in the training dataset (i. e. , background), "stratified", which generates randomly predictions at probabilities that respect the class distribution of the training dataset (i. e. , 1/3 cyclone, 2/3 background), and "uniform", which generates predictions uniformly at random background or cyclone with equal probability.
In our study, the CNN performs significantly better than simple models as shown in Fig. 7. By plotting the values of the metrics in Fig. 8 , we can see that although very close, the performances of the CNN are grouped according to their original dataset (MERRA-2 and ERA5) and that the performances of these two groups seem significantly different. In order to have an objective confirmation, we chose to compare the values of the metrics using the Kruskal-Wallis test, as the distributions of the metric values are not mostly normal (see Table  A2 ). Table  A3  summarizes the pairwise comparison of the metric performances according to the image set used and confirms our interpretation of Fig. 8. This experiment tells us that the difference between the metric values computed from the same dataset, interpolated and not interpolated, can be attributed to randomness, whereas the metrics computed from different dataset are quite distinct.
Cross-comparisons In this section we are interested in the values of the metrics of the CNN trained on one image set and tested on the other image set with the same properties (image resolution and size). In the same way as the previous experiment, we computed the expectation and standard deviation for each of the metrics, given in Table  A4 , and then compared the performance obtained previously (training and testing with the same image set) with these values (training and testing with a different image set). Figure  9  gives the graphical representation and Table  A5  gives the result of the Kruskal-Wallis tests. This experiment shows us that regardless of the resolution and the dataset used for model training, the metric values are statistically distinct and the value of the metrics evaluated on the ERA5 dataset is greater than that evaluated on the MERRA-2 dataset.
Misclassified images Following the comparison of the metrics, we took a closer look at the metadata of the images misclassified by the CNN. Table  A6  in the Appendix summarizes the number of false alarms for each combination of training and testing datasets discussed in Sect. 4. 1. We studied the metadata of the failed predictions that are common to all training/testing datasets so as to limit the study to the most significant cases. We also contextualize the misclassified images in the HURDAT2 time series. There is a total of 15 false alarms in common, i. e. , seven false positives (background images wrongly classified as cyclones) and eight false negatives (cyclone images wrongly classified as background). However, we found that the false negatives and the false positives were generated from the tracks of the same cyclones.
False positives First, we studied the false positives and have listed them in Table  A7. For each image, we gave their HURDAT2 status (see Table  A1 ) as well as the average probability given by the CNN for each dataset (mean probability column). For each of the false positives, we verified if there was a cyclone close in the past and in the future by querying the HURDAT2 https://doi. org/10. 5194/gmd-15-7051-2022 Geosci. Model Dev. , 15, 7051-7073, 2022 database, and indicated the number of hours that separate them from a referenced cyclone (status HU or TS; respectively, the past and future columns). What we can already observe is the high value of the mean probability and its low standard deviation: the CNN is wrong with high confidence for these images whatever the dataset used, which confirms the relevance of the failed predictions in common.
False negative We have listed the single example of false negative that the training/testing datasets have in common in Table  A8 , and we give a graphical example in Fig. 11. The image refers to a cyclone, whose status is TS, and we give its mean probability and standard deviation computed by the CNN for each dataset. We computed the lifetime of cyclonic activity near the geographical area of this image, as previously by querying the HURDAT2 database and indicated the number of hours that separate this image from the first track of a cyclone in the area. We observe that the probability is very low, which means that the CNN is wrong with high confidence, and the low standard deviation of this probability means that this false negative classification is relevant for all the combinations of training/testing datasets.
The choice of interpolation method is particularly relevant in the case of an intercomparison of data from multiple sources with different native resolutions, such as is the case in climate model intercomparison studies. Our cross-comparison experiments have shown that applying an ERA5-trained CNN to MERRA-2 images works better than applying a MERRA-2-trained CNN to ERA5 images, which suggests that ERA5 has a larger information content in the framework of our CNN. This is also consistent with the findings of  Malakar et al. (2020) , who analyzed the error in the location of the center, maximum winds, and minimum pressure at sea level in six meteorological reanalyses including ERA5 and MERRA-2 for the evolution of 28 TCs occurring between 2006 and 2015 over the north Indian Ocean, with respect to the observations of the Indian Meteorological Department (IMD).
The authors of this study show, among other things, that the ERA5 dataset captures the evolution of these TCs in a more realistic way than MERRA-2 (i. e. , smaller errors in the previous variables). They also show that ERA5 and MERRA-2 can capture the intensity of the TCs from the depression stage to the very severe cyclonic storm stage but not from the extremely severe cyclonic storm stage for which the intensity of the TCs is underestimated. However, they conclude that of the six datasets, ERA5 provides the best representation of the TC structure in terms of intensity. Finally, the study published by  Hodges et al. (2017)  shows that 95 % of the Northern Hemisphere TC tracks, from the IBTrACS database that includes HURDAT2, are present in MERRA-2. Unfortunately, this study does not include ERA5.
It also confirms the underestimation of cyclone intensity in MERRA-2 compared to observations. Some transfer learning experiments would also be interesting to conduct. For example, instead of training the CNN with randomly initialized weight values, training the CNN on one image set with weight values initialized with those of the CNN trained on the other image set with the same properties could improve the performance of the CNN. Data augmentation (especially geometric transformations;  Shorten and Khoshgoftaar, 2019)  and model regularization techniques (e. g. , weight decay, batch normalization, dropout) are proven ways to improve the robustness of a CNN trained with a dataset of limited size. Our dataset contains 26 954 images, which is relatively small compared to the size of datasets encountered in many computer vision applications (for instance, Imagenet contains more than 14 million images).
However, using these techniques was not justified for our study because the performance of our CNN without data augmentation is already very high (AUC and AUPRC are over 0. 99). Such techniques could however become very relevant in future work when we seek to detect TCs in climate model simulations with a CNN trained on a reanalysis dataset. Indeed different climate models may simulate TCs imperfectly, and there is probably some value in offering a larger variety of TC structures to the training dataset. It is expected that the simulation of TCs increases in quality with the climate model resolution  (Strachan et al. , 2013) , and climate models running at resolutions of 10 to 50 km are now commonplace. Likewise we would need to augment the number of images with very intense TCs or TCs migrating outside their usual domains because there are indications that such situations may become more frequent with global warming (as presented in the introduction), and we want to ensure these can be detected adequately in climate simulations.
Data augmentation techniques that introduce deformed images into the training datasets could help to increase the robustness of the CNN in these situations. Finally, pixel attribution experiments (saliency maps) should give us the importance of each variable, with hints at a possible reduction in their number or at the use of composite variables such as vorticity. These experiments could also give explanations of misclassified images. Occlusion-perturbation-based methods like local surrogate (LIME;  Ribeiro et al. , 2016) , Shapley values (SHAP; Lundberg and Lee, 2017), and gradient-based methods like Grad-CAM  (Selvaraju et al. , 2017 ) should be resourceful.
Conclusions In this study, we have adapted and tested a CNN for the classification of images according to the presence or absence of tropical cyclones. The image sets for training and tests were built from the ERA5 and MERRA-2 reanalyses with labels derived from the HURDAT2 dataset. We have paid a Accuracy ERA5 32px@0. 25/same ERA5 16px@0. 5/same 9. 173333 × 10 -1 True ERA5 32px@0. 25/same MERRA-2 32px@0. 25/same 8. 451539 × 10 -31 False ERA5 32px@0. 25/same MERRA-2 16px@0. 5/same 2. 367641 × 10 -33 False ERA5 16px@0. 5/same MERRA-2 32px@0. 25/same 9. 721777 × 10 -29 False ERA5 16px@0. 5/same MERRA-2 16px@0. 5/same 9. 251784 × 10 -31 False MERRA-2 32px@0. 25/same MERRA-2 16px@0. 5/same 5. 506329 × 10 -1 True AUC ERA5 32px@0. 25/same ERA5 16px@0. 5/same 5. 243858 × 10 -1 True ERA5 32px@0. 25/same MERRA-2 32px@0.
Metric Training dataset Test dataset Estimated Estimated Shapiro p Shapiro p mean SD on means on all Accuracy ERA5 32px@0. 25 MERRA-2 32px@0. 25 0. 977677 0. 003182 0. 226578 0. 017054 ERA5 16px@0. 5 MERRA-2 16px@0. 5 0. 977721 0. 002321 0. 786720 0. 013096 MERRA-2 32px@0. 25 ERA5 32px@0. 25 0. 986523 0. 002604 0. 073408 0. 193609 MERRA-2 16px@0. 5 ERA5 16px@0. 5 0. 986655 0. 002943 0. 540750 0. 019739 AUC ERA5 32px@0. 25 MERRA-2 32px@0. 25 0. 995235 0. 001789 0. 762134 5. 846574 × 10 -9 ERA5 16px@0. 5 MERRA-2 16px@0. 5 0. 995413 0. 001208 0. 871147 2. 561682 × 10 -12 MERRA-2 32px@0. 25 ERA5 32px@0. 25 0. 998357 0. 000803 0. 719871 1. 581214 × 10 -12 MERRA-2 16px@0. 5 ERA5 16px@0. 5 0. 998323 0. 000929 0. 215594 5. 414141 × 10 -9 AUPRC ERA5 32px@0. 25 MERRA-2 32px@0. 25 0. 993296 0. 001898 0.
Metric Training/test dataset Training/test dataset Kruskal p value Comparable Accuracy ERA5 32px@0. 25/same ERA5 32px@0. 25/MERRA-2 32px@0. 25 4. 857791 × 10 -47 False ERA5 16px@0. 5/same ERA5 16px@0. 5/MERRA-2 16px@0. 5 1. 005262 × 10 -44 False MERRA-2 32px@0. 25/same MERRA-2 32px@0. 25/ERA5 32px@0. 25 1. 825678 × 10 -11 False MERRA-2 16px@0. 5/same MERRA-2 16px@0. 5/ERA5 16px@0. 5 1. 434371 × 10 -14 False AUC ERA5 32px@0. 25/same ERA5 32px@0. 25/MERRA-2 32px@0. 25 4. 011721 × 10 -46 False ERA5 16px@0. 5/same ERA5 16px@0. 5/MERRA-2 16px@0. 5 2. 698731 × 10 -43 False MERRA-2 32px@0. 25/same MERRA-2 32px@0. 25/ERA5 32px@0. 25 7. 645103 × 10 -9 False MERRA-2 16px@0. 5/same MERRA-2 16px@0. 5/ERA5 16px@0. 5 2. 074129 × 10 -12 False AUPRC ERA5 32px@0. 25/same ERA5 32px@0. 25/MERRA-2 32px@0. 25 1.
640760 × 10 -48 False ERA5 16px@0. 5/same ERA5 16px@0. 5/MERRA-2 16px@0. 5 3. 619129 × 10 -47 False MERRA-2 32px@0. 25/same MERRA-2 32px@0. 25/ERA5 32px@0. 25 3. 864565 × 10 -12 False MERRA-2 16px@0. 5/same MERRA-2 16px@0. 5/ERA5 16px@0. 5 1. 332830 × 10 -16 False Table  A8. The single TC-containing image wrongly classified as background (false negative) for all combinations of training/testing datasets. The status of the image according to HURDAT2 and the probability of the classification (with its standard deviation across the combinations of training/testing datasets) are indicated. The columns past and future reflect the cyclonic activity in the geographical area of the image, i. e. , the temporal distance to the first and the last tracks of the cyclone (and their HURDAT2 status). versité, Ecole Polytechnique, and CNES as well as through national and international grants.