SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Asanobu Kitamoto $^{1,2}$  Jared Hwang $^{3,1}$  Bastien Vuillod $^{4,1}$  Lucas Gautier $^{5,1}$
Yingtao Tian $^{6}$  Tarin Clanuwat $^{6}$
$^{1}$  National Institute of Informatics, Japan
$^{2}$  Typhoon Science and Technology Research Center, Yokohama National University, Japan
<sup>3</sup> University of Southern California, USA
4 Grenoble-INP, Ensimag, France
<sup>5</sup> Université Clermont Auvergne, ISIMA, France
This paper presents the official release of the Digital Typhoon dataset, the longest typhoon satellite image dataset for  $40+$  years aimed at benchmarking machine learning models for long-term spatio-temporal data. To build the dataset, we developed a workflow to create an infrared typhoon-centered image for cropping using Lambert azimuthal equal-area projection referring to the best track data. We also address data quality issues such as inter-satellite calibration to create a homogeneous dataset. To take advantage of the dataset, we organized machine learning tasks by the types and targets of inference, with other tasks for meteorological analysis, societal impact, and climate change. The benchmarking results on the analysis, forecasting, and reanalysis for the intensity suggest that the dataset is challenging for recent deep learning models, due to many choices that affect the performance of various models. This dataset reduces the barrier for machine learning researchers to meet large-scale real-world events called tropical cyclones and develop machine learning models that may contribute to advancing scientific knowledge on tropical cyclones as well as solving societal and sustainability issues such as disaster reduction and climate change. The dataset is publicly available at http://agora.ex.nii.ac.jp/digital-typhoon/dataset/ and https://github.com/kitamoto-lab/digital-typhoon/.
Tropical cyclones, also known as typhoons and hurricanes in certain regions, have been the critical target of research due to their substantial societal impact [11]. To reduce the impact of tropical cyclones, the meteorological community, along with other earth science communities, has been developing both a theoretical and an empirical understanding of tropical cyclones through efforts such as advancing satellite remote sensing and atmospheric simulation models of higher spatial, temporal, and spectral resolutions for better analysis and forecasting.
Meteorologists have also developed an empirical method, known as the Dvorak technique [10, 54], to estimate the intensity of a tropical cyclone based on time-series observation data collected from worldwide ground sensor networks, meteorological satellites, and reconnaissance flights. This technique consists of a manual procedure to estimate tropical cyclone intensity based on the cloud
Figure 1: Overview of the Digital Typhoon dataset.
patterns of satellite images and a temporal model for intensity change. The method was originally developed in the United States in the 1970s and later adopted by meteorological agencies worldwide to become the standard procedure. However, experts are aware of its heuristic and subjective nature, as it relies on empirical, rather than theoretical, human interpretation of observation data. Solutions to this problem include more objective and automated versions of the Dvorak technique[39, 38] and a citizen science project to take advantage of collective intelligence [15].
It is clear that the Dvorak technique naturally fits into the machine learning framework by using images as input and intensity values as output. Hence there is a growing interest in both the machine learning community [42, 8, 36, 33] and the meteorology community [18, 5] to take advantage of the big data of tropical cyclones for developing data-driven approaches. One of the authors, Asanobu Kitamoto, started the Digital Typhoon project in 1999 with the aim of applying machine learning to typhoon analysis and forecasting [21, 26]. The first step was to develop a homogeneous satellite image dataset for machine learning as in Figure 1. The second step was to apply machine learning algorithms available at the time, such as SVM [24], Generative Topographic Mapping [22], and content-based image retrieval[23], which is later evolved into deep learning-based models for classification and regression tasks [46, 41], combined with fisheye preprocessing [16]. The third step was to release the website "Digital Typhoon" in 2003 for browsing and searching datasets [25]. The remaining problem was the lack of public datasets for machine learning. There have been attempts to download the dataset via scraping of the website (e.g. [53]), but the dataset created in this way is of lower quality.
Here we introduce the Digital Typhoon dataset, the longest typhoon satellite image dataset. This dataset reduces the burden of researchers to start machine learning on tropical cyclones without solid domain knowledge of meteorology and satellite remote sensing. We also illustrate the variety of tasks so that researchers can concentrate on building and evaluating machine-learning models.
2.1 Track Datasets
The track data includes the 'annotation' of tropical cyclones, such as location, intensity, and wind circles, based on the interpretation of meteorological experts following the established procedure (e.g. Dvorak Technique). The best estimate, obtained from a retrospective analysis after collecting all the information from the start to the end of life, is called the best track dataset.
The Digital Typhoon dataset targets the Western North Pacific basin, and the Japan Meteorological Agency (JMA) is designated as the regional center to maintain the best track dataset. Globally, the International Best Track Archive for Climate Stewardship (IBTrACS) [28] collects the best track from meteorological agencies worldwide and creates a comprehensive track dataset since 1842.
IBTrACS shows an interesting variation of the best track; namely the location and intensity of the same tropical cyclone show discrepancies across meteorological agencies [48]. This fact suggests
(a) Number of images for each grade
(b) Transition matrix for the grade
(c) Number of typhoons by the number of images
Figure 2: Visualization of statistics of the Digital Typhoon dataset.
that the interpretation of the observation data is not unique, or not the ground truth in a strict sense. Nonetheless, we regard the best track as the ground truth for most machine learning tasks, because it is the best estimate available. In a reanalysis task, however, we could critically evaluate the quality of the best track [17].
2.2 Image Datasets
The image dataset has information about the spatial distribution of physical properties such as cloud patterns as grid data. The observation dataset [27, 30] is derived from sensor observation that measures the physical properties of the atmosphere, while the simulation dataset, both typhoon-related [37] and the global atmosphere [44, 1, 3, 43], is generated as the representation of the atmosphere in a simulation model. Observation datasets and simulation datasets are linked through data assimilation, which is a statistical method to integrate observation datasets into a simulation model.
The Digital Typhoon dataset is an observation dataset, and it offers a richer detail of tropical cyclones with higher temporal and spatial resolutions than the simulation dataset. In addition, data quality issues in the observation dataset, such as sensor noise, missing data, and long-term sensor calibration, are handled properly so that machine learning models are not significantly affected by those issues.
3 Digital Typhoon Dataset
3.1 Dataset Overview
The Digital Typhoon dataset is created from the comprehensive satellite image archive of the Japanese geostationary satellite series, Himawari, from Himawari-1 to Himawari-9. Although those images are not copyrighted, some data are not accessible for free, and old satellite images have old formats for which open-source parsers are difficult to find. Hence we developed our own parsers for all generations of satellites, and the workflow to create typhoon-centered images by referring to the best track, as shown in Figure 1.
Using this workflow, we created the Digital Typhoon dataset by integrating metadata and images. The metadata contains hourly best-track data with additional information about the file name and each image's quality. The formatting of the best track data aligns with the original best track data sourced from the JMA. On the other hand, the images feature a 2D array of brightness temperatures around the typhoon's center, formatted in HDF5.
As a result, the dataset comprises a total of 1,099 typhoons and 189,364 images. Figure 2 visualizes some of the statistics of the dataset. It is a complete record of typhoons occurring in the Western North Pacific region (ranging from 100 to 180 degrees east of the northern hemisphere), from the 1978 season through the 2022 season, with missing typhoons in 1979 and 1980 due to the unavailability of satellite data. The length of the dataset, spanning 44 typhoon seasons (years), is the longest typhoon image dataset. We call it the longest dataset because Japanese geostationary satellite images for typhoons before 1978 were lost forever, and our dataset went back to the oldest satellite image preserved. Hence it provides a unique opportunity to challenge long-term datasets.
Table 1: Comparison between the Digital Typhoon dataset and the HURSAT dataset.
Digital Typhoon dataset	HURSAT dataset
Temporal coverage	1978-2022 (present)	1978-2015
Temporal resolution	one hour	three hours
Target satellites	Himawari	SMS, GOES, Meteosat, Himawari, FY2
Spatial coverage	Western North Pacific basin	All basins (Global)
Spatial resolution	5km	8km
Image coverage	512×512 pixels (1250km from the center)	301×301 pixels (1100km from the center)
Spectral coverage	infrared (others on the Website)	visible, infrared, water vapor, near IR, split window
Map projection	Azimuthal equal-area projection	Equirectangular projection
Calibration	Recalibration	ISCCP
Data format	HDF5	NetCDF
Best track	Japan Meteorological Agency	IBTrACS
Dataset browsing	Digital Typhoon website	Download only
The Digital Typhoon dataset can reduce the burden of machine learning researchers to study tropical cyclones. First, it opens up access to tropical cyclone data processed from long-term satellite data. Second, it offers a homogeneous dataset created by the image processing workflow based on expertise in meteorology and satellite remote sensing. Third, massive computations to process hundreds of terabytes of original satellite data to create a machine-learning dataset are not necessary. A comprehensive explanation of the workflow for the creation of the dataset is provided in the Appendix.
The Digital Typhoon dataset is available at the official page http://agna.ex.nii.ac.jp/digital-typhoon/dataset/ with an open data license, namely the Creative Commons Attribution 4.0 International (CC BY 4.0) License.
3.2 Comparison with the HURSAT Dataset
Among satellite image datasets of tropical cyclones, Hurricane Satellite Data (HURSAT) dataset [27, 30] from The National Oceanic and Atmospheric Administration (NOAA) is the most notable dataset in size and coverage. Table 1 provides a comparative summary of the Digital Typhoon and HURSAT datasets. There are distinct variations between the two as enumerated below.
Temporal coverage The Digital Typhoon dataset is continually updated, and is the longest tropical cyclone image dataset worldwide. On the other hand, the HURSAT dataset stopped updating in 2015.
Temporal resolution The Digital Typhoon dataset has a temporal resolution of one hour which is higher than the HURSAT dataset's three-hour resolution. A high-frequency change such as rapid intensification is more sensitive to temporal resolution.
Spatial coverage The Digital Typhoon dataset specifically targets the Western North Pacific basin, whereas the HURSAT dataset encompasses all basins.
Spatial resolution The Digital Typhoon dataset possesses a spatial resolution of approximately 5km, superior to the HURSAT dataset's roughly 8km (0.07 degree). A small-scale structure such as the eye of a tropical cyclone is more sensitive to spatial resolution.
Spectral coverage The Digital Typhoon dataset incorporates the infrared (IR) channel, while the HURSAT dataset has more channels. It should be noted, however, that the Digital Typhoon website has the same spectral coverage, and the dataset can be easily extended to cover these channels.
Map projection The Digital Typhoon dataset utilizes the Lambert azimuthal equal-area projection, maintaining the spherical shape of the tropical cyclone, while the HURSAT dataset employs the
equirectangular (lat/long) grid, causing shape distortion in higher latitude or peripheral areas. Figure 1 shows an example of distortion when a typhoon is observed in the north.
Dataset browsing The Digital Typhoon dataset can be browsed via the Digital Typhoon website, which offers additional data. In contrast, the HURSAT dataset is solely available for download.
3.3 Design Choices
The dataset has several design choices, such as spectral coverage, spatial resolution, temporal resolution, and spatial coverage. In the following, we explain the reasons behind our choices.
Spectral coverage The current dataset includes only the Infrared channel (IR1) (wavelength of around  $11\mu \mathrm{m}$ ) but does not include any other channels available on the Digital Typhoon website. The following is the summary of the availability of each channel on the website.
- IR1 (infrared): the data has been available since the beginning (1978).
- VIS (visible): the data has been available since the beginning (1978), but images from early satellites were too noisy and not appropriate for a machine learning dataset. In addition, the visible channel is meaningful only during the daytime.
- IR2 (infrared) and WV (water vapor): the data has been available since 1995 (Himawari-5).
- NIR (near infrared) and other channels: the data has been available since 2005 (2nd generation) or 2015 (3rd generation).
As summarized, the IR1 is the only channel that is the longest and with fewer data quality issues, and this is the reason we included only the IR1 channel in our first version of the dataset. Future inclusion of multispectral data may offer additional tasks such as multispectral classification and regression.
Spatial resolution The spatial resolution of about  $5\mathrm{km}$  per pixel reflects the spatial resolution of the IR1 channel for the first-generation satellites from Himawari-1 to Himawari-5. This resolution has improved to  $4\mathrm{km}$  for the second generation and  $2\mathrm{km}$  for the third generation. In spite of these progresses in technology, we chose a  $5\mathrm{km}$  resolution because it is the best choice to create a long-term homogeneous dataset. An interesting task in the future is to transfer a machine-learning model from long-term lower-resolution datasets to short-term higher-resolution datasets so that we can take advantage of recent technology for better forecasting.
Temporal resolution The temporal resolution of one hour reflects the temporal resolution of one hour for some of the first-generation satellites after Himawari-3. From Himawari-1 to Himawari-2, the temporal resolution was more than one hour, or typically every three hours. For this reason, the data before 1987 has many missing data points as an hourly dataset. This resolution has improved to 30 minutes for the second generation and 10 minutes for the third generation. In spite of these progresses in technology, we chose one hour because it is a representative interval for many types of meteorological observations.
Spatial coverage The current dataset only covers the Western North Pacific basin in the northern hemisphere (NH), but the Digital Typhoon website offers the same types of images for the southern hemisphere (SH) in the Australian basin using the best track from the Bureau of Meteorology, Australia. Here an interesting question is how a model trained in NH can be transferred to SH. From a meteorological point of view, tropical cyclones in various basins are considered the same meteorological phenomena, so theoretically, the dataset can be created similarly, and machine learning results are transferable. However, we also need to consider many details that may have an impact on the actual results, such as different quality of the best track data, and different sensor characteristics and calibration methods for different satellites. A future version of our datasets and benchmarks may address these issues.
4 Machine Learning Tasks
The Digital Typhoon dataset serves two important roles. First, it offers a practical real-world dataset and tasks for the machine learning community to explore new models and solutions. Second, it
provides a tool for meteorologists to apply data-driven approaches in studying tropical cyclones. The following is a summary of tasks in multiple dimensions. Other lists of tasks can be found in the review [7, 55, 45].
4.1 Types of Inference
Analysis The task is to estimate current values using the current and past data. For instance, estimating the intensity of a typhoon falls into an analysis task, as it produces information about the typhoon's intensity using both current and past data. Supervised learning within this task can be further categorized into either a classification task or a regression task, contingent on whether the target variable is categorical or numerical. Additionally, unsupervised tasks can be designed for clustering or identifying typhoons with similar characteristics.
Forecasting The task is to produce future predictions based on current and past data. The forecasts can be evaluated with the actual outcomes from the real event which become available over time. The forecasting task has a sub-task called nowcasting, aimed at making short-term forecasting spanning several hours using data-driven extrapolation. Note, however, that weather forecasting is theoretically constrained by the atmosphere's chaotic nature, which states that a minor difference in initial conditions can escalate over time.
We call this task 'forecasting' instead of 'prediction' because prediction is ambiguous in machine learning. In meteorology, prediction is strictly used to mean future values, but in machine learning prediction could mean the output of a machine learning model without temporal dimension. To avoid confusion across disciplines, we use forecasting throughout the paper.
Reanalysis The task is to produce the best estimate given all obtainable data. This task is especially relevant to producing a uniform dataset spanning a long period of time, such as detecting trends in tropical cyclone activity to study the effects of climate change. As addressed in Section 2.1, the best track dataset may contain errors due to technological limitations or inconsistencies from different human experts. Machine learning can potentially aid in evaluating the quality of annotated data.
4.2 Targets of Inference
Intensity The task makes inferences on the strength and size of a typhoon. The categorical grade is used to classify both the strength and type of a tropical cyclone. A classification task uses these grades as target variables. On the other hand, the intensity of tropical cyclones is measured numerically by central pressure and maximum sustained wind. An intensity regression task uses either pressure or wind as the target variable. In addition, the metadata includes the radius of the strong wind circle that represents the size of a tropical cyclone, so we can also design a regression task for size using the radius as the target variable.
Track The task makes inferences on the geographical location of a typhoon. The cyclone's center, as estimated by human experts, is represented by latitude and longitude coordinates with a precision of 0.1 degrees. A regression task for predicting the typhoon's location uses these latitude and longitude coordinates as target variables.
Formation The task makes inferences on the birth of a tropical cyclone, which typically occurs in tropical regions. Among the numerous cloud clusters actively evolving in tropical regions, determining which one will evolve into a tropical cyclone presents a challenging forecasting task, making it a target for machine learning applications [6].
Transition The task makes inferences on the transition from a tropical cyclone to an extra-tropical cyclone, which typically occurs in mid-latitude regions. Two types of cyclones are conceptually distinct from a meteorological perspective, but as a natural phenomenon, they are continuous. The data-driven modeling of a continuous transition process connecting two discrete concepts is a machine-learning task.
Table 2: The statistics of target values.
Target value	Range	Mean	Standard deviation
Central pressure	870-1018 (hPa)	983.8	22.5
Maximum sustained wind	35-140 (knots)	59.2	19.8
4.3 Meteorological Analysis
Machine learning can also be applied to analyze meteorological events on tropical cyclones, such as rapid intensification [2], eyewall replacement [14], and overshooting cloud tops [20]. These events may be linked with the forecasting of tropical cyclones, yet their underlying mechanisms are not entirely understood. Data-driven methodologies could potentially provide insights that contribute to the development of a novel theoretical framework for understanding these phenomena.
4.4 Analysis for Societal Impact
The Digital Typhoon dataset represents the atmospheric observation of a tropical cyclone, but its societal impacts are measured by different sources and modalities. For example, hazards are measured by heavy rainfall or strong winds, disasters are measured by landslides and flooding, and damages are measured by human casualties and financial loss. To construct a machine learning model to analyze and forecast the societal impact, real-world datasets from many sources should be integrated with meteorological datasets. This would enable a more comprehensive understanding of the full range of impacts arising from tropical cyclones.
4.5 Analysis for Climate Change
Understanding how a long-term tropical cyclone activity is impacted by climate change is a crucial topic in society [32, 29, 31, 47]. Technological and methodological evolution that occurred during the  $40+$  years lifespan introduces many types of biases in the dataset. While certain biases may be removed by sensor calibration, others are harder to detect such as annotation errors by human experts. The reanalysis of historical data and the creation of a homogeneous dataset can contribute to advancing our knowledge of the relationship between tropical cyclones and climate change.
5 Benchmarks
5.1 Overview of Benchmarks
Task Machine learning tasks can be combined to create benchmarks for machine learning. We propose three benchmarks, 1) Analysis, 2) Forecasting, and 3) Reanalysis of the intensity of typhoons. The following summarizes some of the technical choices for benchmarking.
Data splitting In meteorological time series, data are auto-correlated and one has to be careful how to split the data before starting to train a model [49]. At least, a random split for the image level must not be used to avoid overestimating the performance due to data leakage in the same typhoon sequence. Our assumption is that every sequence is independent, and we do not have to consider any leakage across sequences. So, as long as each sequence is treated as atomic when splitting the dataset, there is no limitation to using the entire dataset. Hence we apply random splits to the sequence level (split-by-sequence) or the season level (split-by-season). More complex splits can be designed, such as split by satellite generations (1978-2004, 2005-2014, 2015-2022). These designed splits are especially useful for the reanalysis task in Section 5.4.