SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Performance metric The following benchmarks evaluate the performance by the absolute error of target values because this is easier for domain experts to understand the result. However, for machine learning experts, the relative error of target values is more intuitive. Instead of showing absolute and relative errors for each benchmark, we summarize the statistics of target values so that relative errors can be roughly estimated. For example, the best result of  $10.06 \pm 0.09$  hPa RMSE in Table 3 is less than one standard deviation of error.
Table 3: The result of the pressure regression task for two architectures and three types of input.
RMSE (hPa)	Full (512 × 512)	Resized (224 × 224)	Cropped (224 × 224)
ResNet18	10.51 (±0.11)	10.47(±0.20)	10.06 (±0.09)
ResNet50	11.12 (±0.41)	11.63 (±0.35)	10.09 (±0.04)
Table 4: The result of the wind regression task for two architectures and three types of input.
RMSE (kt)	Full (512 × 512)	Resized (224 × 224)	Cropped (224 × 224)
ResNet18	10.21 (±0.19)	10.09 (±0.08)	9.25 (±0.25)
ResNet50	10.05 (±0.26)	10.21 (±0.14)	9.13 (±0.11)
Software and hardware To perform the benchmarks, we developed a Python-based software library pyphoon2, downloadable from https://github.com/kitamoto-lab/digital-typhoon/. pyphoon2 comes with a data loader and components to help build machine learning pipelines. All the experiments were performed on the internal cluster with 6 GPUs consisting of NVIDIA Quadro RTX 6000, NVIDIA Quadro RTX 8000, and NVIDIA Quadro RTX A6000.
5.2 Analysis for the Intensity
We propose classification tasks, which take an image as input and estimate grade as output, and regression tasks, which take an image as input and estimate a pressure or wind value as output. In the JMA best track, grades 3, 4, and 5 denote a tropical cyclone, among which grade 5 is the most intense according to the maximum sustained wind. Grade 2 signifies a tropical depression, a type of cyclone weaker than a tropical cyclone. Moreover, grade 6 corresponds to an extra-tropical cyclone, a type of cyclone having a different structure from a tropical cyclone. Central pressure in hectopascal (hPa) is recorded for all grades, while the maximum sustained wind in knot (kt) is recorded only for grades 3, 4, and 5. In the following, we describe the result of the regression task, and the result of the classification task is described in the appendix.
We explored three types of comparisons. First, we compared three architectures, namely VGG [51], ResNet [13] and Vision Transformer [9]. Second, we compared models trained on 1) full-resolution images  $(512 \times 512)$ , 2) resized images  $(224 \times 224)$ , and 3) cropped images  $(224 \times 224)$ . In 2), the full region of the image is resized, while in 3), the central region of the image is cropped without resizing. The latter is inspired by the Dvorak technique, which focuses on many relevant image features found around the typhoon center. Third, we compared two target values, namely pressure, and wind.
We used the TorchVision [35] ResNet18 and ResNet50 models with a learning rate (LR) of  $10^{-4}$ , batch size of 16, and for 50 epochs. An 80/20 train/test split by sequence was used. The ResNet18 and ResNet50 models were trained five and two times respectively. To evaluate, we measured the root mean square error (RMSE) of the prediction from ground truth and their standard deviations  $(\pm \mathrm{std})$
Table 3 and Table 4 summarize the results. Firstly, ResNet50 yielded similar results to ResNet18. Secondly, cropping the images around the typhoon center yielded a lower RMSE than other choices, indicating that cropping is better than resizing in preserving features around the typhoon center, or removing non-relevant features far from the center. Training a model on the full images did not perform well due to their larger number of pixels. Furthermore, Figure 3 illustrates that regression performs better for weaker typhoons, but worse for stronger typhoons.
5.3 Forecasting for the Intensity
Our previous work used Recurrent Neural Network (RNN) to forecast the pressure directly from images and showed comparable performance with SHIPS [41]. In this paper, we chose another approach using a convolutional LSTM [50] to predict the next  $n$  image frames of a typhoon given the previous 12 image frames and analyze the pressure from the predicted image. We adapted an implementation [40], and used a 3-layer ConvLSTM with 128 hidden dimensions. Due to resource limitations, we used  $128 \times 128$  downsampled images from only the first 24 hours of a given typhoon.
(a) Plots for wind regression by ResNet18 for cropped images.
(b) Plots for pressure regression by ResNet18 for cropped images.
Figure 3: Prediction plots for pressure and wind regressions.
Figure 4: Results of image forecasting by ConvLSTM.
To forecast  $n$  hours into the future (starting at  $t = 1$ ), the 12 preceding frames ( $t = [-11,0]$ ) are passed into the model, which outputs a single image serving as its forecast at  $t = 1$ . Then, images from  $t = [-10,1]$ , including the predicted image, are passed back into the model to get the prediction for  $t = 2$ . This process is repeated  $n$  times to forecast  $n$  hours into the future. As a result, Figure 4 shows that the first predicted frame is perceptively blurred and rapidly deteriorates as  $t$  advances.
We then trained a ResNet18 model on predicted images, as well as the first 24 images of every typhoon, to predict the pressure given a  $128 \times 128$  image. As a result, Table 5 shows that the model produces a larger RMSE and error as  $t$  advances due to the blur of predicted images. A future adaptation may be to train both the ConvLSTM and ResNet in a black box, such that the loss is minimized by image reproduction and pressure prediction.
Both models were trained on the same 80/20 train/test split by sequence. The ConvLSTM was trained once for 230 epochs with a starting LR of  $10^{-4}$ , and used a CosineAnnealing scheduler [34] with 100 steps. The ResNet model used a modified first convolutional layer with a kernel and stride size of (2, 2) and (1, 1). It was trained five times with an LR of  $10^{-5}$  for 34 epochs. These hyperparameters were chosen as they produced more consistent results given the smaller image sizes.
5.4 Reanalysis for the Intensity
The goal of this paper is to create a homogeneous long-term dataset, and the purpose of the reanalysis task is to identify biases and inconsistencies in the dataset due to factors such as technological evolution arising from satellite sensors, or methodological evolution arising from the improvement of the Dvorak technique to annotate tropical cyclones. One approach to this challenge is to design special data splits to analyze historical factors.
Table 5: Results of pressure forecasting for 12-hours by ResNet18 (values in hPa).
t	1	2	3	6	12
RMSE	10.24 ± 0.73	10.52 ± 0.79	11.00 ± 0.87	12.10 ± 0.85	14.69 ± 0.91
Table 6: The results of regression tasks for each satellite generation (values in hPa).
RMSE	Train the First	Train the Second	Train the Third
Test the First	10.04 (±0.17)	9.92 (±0.09)	10.03 (±0.10)
Test the Second	12.80 (±0.19)	11.05 (±0.10)	11.17 (±0.10)
Test the Third	10.34 (±0.17)	10.03 (±0.08)	9.94 (±0.16)
Our previous work studied this task by training the model using recent data and testing on past data to analyze the trend of model performance, indicating that old satellite data may have different characteristics [41]. In this paper, we split the dataset into three buckets by satellite generations, namely the first generation (1978-2004), the second generation (2005-2014), and the third generation (2015-2022), and train and test a ResNet18 model for the regression task.
Input images were resized to  $224 \times 224$  in the same way as the analysis task. 208 sequences (the size of the smallest generation) were then randomly sampled from each generation five times and split into 80/20 train/test sets. A ResNet18 model was trained on each bucket, each for 101 epochs with a batch size of 16 and a learning rate of  $10^{-4}$ ; these parameters were chosen for their consistent results. Each of the three models was then tested on the test set of each bucket, such that a model trained on the first bucket was tested on the first, second, and third buckets.
Table 6 shows that all three models performed roughly similarly on all three buckets, and no dataset bias was immediately reflected in the quality of the models. An expansion on the reanalysis task experiment we performed is described in the Appendix.
5.5 Comparison with Other Approaches
Machine learning is not the only approach for data-driven analysis and forecasting of tropical cyclones. For the analysis of intensity, the Dvorak technique has been the most popular method among meteorologists. In addition, for the forecasting of intensity, computational approaches represent a typhoon in a simulation model and compute the future based on the theory of the atmosphere. This approach, however, has limitations due to spatial and temporal resolutions, and intensity forecasting is still considered a difficult challenge. Instead, meteorologists have developed empirical methods, such as SHIPS [12, 56] with linear regression on hand-crafted meteorological features, or a similar approach using XGBoost [4]. This paper focused on the comparison of machine learning models, but the real challenge for domain experts is comparing not only machine learning approaches but also computational, empirical, or manual approaches in the context of real-world solutions for tropical cyclones, such as disaster reduction. This paper is a starting point for this grand challenge.
We have introduced the Digital Typhoon dataset for machine learning and meteorology communities to promote data-driven research on tropical cyclones. Our dataset offers a unique opportunity to benchmark various types of machine learning models, especially spatio-temporal models for long-term time-series images. A solution is not only valuable for machine learning benchmarking but also has the potential to contribute to advancing scientific knowledge on tropical cyclones as well as solving societal and sustainability issues such as disaster reduction and climate change.
Three of the authors, Jared Hwang, Bastien Vuillod, and Lucas Gautier, have been supported by the international internship program of the National Institute of Informatics.
[50] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai kin Wong, and Wang chun Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting, 2015.
[51] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
[52] Tasuku Tabata, Viju O. John, Rob A. Roebeling, Tim Hewison, and Jörg Schulz. Recalibration of over 35 Years of Infrared and Water Vapor Channel Radiances of the JMA Geostationary Satellites. Remote Sensing, 11(10):1189, January 2019. Number: 10 Publisher: Multidisciplinary Digital Publishing Institute.
[53] Wei Tian, Wei Huangwei, Xiaolong Xu, and Chao Wang. Tropical Cyclone Maximum Wind Estimation from Infrared Satellite Data with Integrated Convolutional Neural Networks. In 2019 International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), pages 575-580, July 2019.
[54] Christopher Velden, Bruce Harper, Frank Wells, John L. Beven, Ray Zehr, Timothy Olander, Max Mayfield, Charles "Chip" Guard, Mark Lander, Roger Edson, Lixion Avila, Andrew Burton, Mike Turk, Akihiro Kikuchi, Adam Christian, Philippe Caroff, and Paul McCrone. The Dvorak Tropical Cyclone Intensity Estimation Technique: A Satellite-Based Method that Has Endured for over 30 Years. Bulletin of the American Meteorological Society, 87(9):1195-1210, September 2006.
[55] Zhen Wang, Jun Zhao, Hong Huang, and Xuezhong Wang. A Review on the Application of Machine Learning Methods in Tropical Cyclone Forecasting. Frontiers in Earth Science, 10, 2022.
[56] Munehiko Yamaguchi, Hiromi Owada, Udai Shimada, Masahiro Sawada, Takeshi Iriguchi, Kate D. Musgrave, and Mark DeMaria. Tropical Cyclone Intensity Prediction in the Western North Pacific Basin Using SHIPS and JMA/GSM. *Sola*, 14:138–143, 2018.