SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wai Tong Chung\*,1, Bassem Akoush\*,1, Pushan Sharma\*,1, Alex Tamkin\*,1, Ki Sung Jung\*,2, Jacqueline H. Chen\*,2, Jack Guo\*,1, Davy Brouzet\*,1, Mohsen Talei\*,3, Bruno Savard\*,4, Alexei Y. Poludnenko\*,5, Matthias Ihme\*,1,6
$^{1}$ Stanford University,  $^{2}$ Sandia National Laboratory,  $^{3}$ University of Melbourne,  $^{4}$ Polytechnique Montréal,  $^{5}$ University of Connecticut,  $^{6}$ SLAC National Accelerator Laboratory
Analysis of compressible turbulent flows is essential for applications related to propulsion, energy generation, and the environment. Here, we present BLASTNet 2.0, a 2.2 TB network-of-datasets containing 744 full-domain samples from 34 high-fidelity direct numerical simulations, which addresses the current limited availability of 3D high-fidelity reacting and non-reacting compressible turbulent flow simulation data. With this data, we benchmark a total of 49 variations of five deep learning approaches for 3D super-resolution – which can be applied for improving scientific imaging, simulations, turbulence models, as well as in computer vision applications. We perform neural scaling analysis on these models to examine the performance of different machine learning (ML) approaches, including two scientific ML techniques. We demonstrate that (i) predictive performance can scale with model size and cost, (ii) architecture matters significantly, especially for smaller models, and (iii) the benefits of physics-based losses can persist with increasing model size. The outcomes of this benchmark study are anticipated to offer insights that can aid the design of 3D super-resolution models, especially for turbulence models, while this data is expected to foster ML methods for a broad range of flow physics applications. This data is publicly available with download links and browsing tools consolidated at https://blastnet.github.io.
In recent years, machine learning (ML) has offered new modeling approaches for natural and engineering sciences. For example, 5 PB of ERA5 data [1] was employed towards GraphCast [2], an ML model that outperformed conventional weather modeling techniques. As such, efforts in curating large datasets for scientific ML have been growing across numerous domains including agricultural science [3], geophysics [4], and biology [5]. In many of these examples, the volume of data can significantly exceed the free limit of open-source repositories such as Kaggle (100 GB) – typically used to store and share language and image data – due to potentially high dimensions within scientific data. As such, significant resources are required for building and maintaining data storage capabilities, either through institutional collaborations [3, 5, 6] or cloud service providers [7].
In previous work [8, 9], we proposed the Bearable Large Accessible Scientific Training Network-of-datasets (BLASTNet), a cost-effective community-driven weakly centralized framework (see Section 3.1) that utilizes Kaggle for increasing access to scientific data, which provided access to 110 full-domain samples from 10 configurations (225 GB) of 3D high-fidelity flow physics direct
*Corresponding authors: {wtchung,mihme}@stanford.edu
numerical simulations (DNS). In this work, we present the updated BLASTNet 2.0 dataset, which is extended to 744 full-domain samples from 34 DNS configurations, as shown in Figure 1. This dataset aims to address limitations in data availability for compressible turbulent non-reacting and reacting flows, which is found in automotive [10, 11], propulsion [12, 13], energy [14, 15], and environmental [16, 17] applications. BLASTNet data has previously been employed for solving ML tasks related to dimensionality reduction, regime classification, and turbulence-chemistry closure modeling [9]. Beyond this, BLASTNet is potentially suited for ML problems involving predictions of physical quantities found in turbulent non-reacting and reacting flows, which can also involve inverse problems [18, 19] and physics discovery [20, 21].
In this work, we demonstrate the utility of BLASTNet data for 3D super-resolution (SR) of turbulent flows. From BLASTNet 2.0, we pre-process DNS data to form the Momentum128 3D SR dataset for benchmarking this task. To this end, we:
- Curate BLASTNet 2.0, a diverse public 3D compressible turbulent flow DNS dataset.
- Benchmark performance and cost of five 3D ML approaches [19, 22-25] for SR with this publicly accessible dataset.
- Show that SR model performance can scale with the logarithm of model size and cost.
- Demonstrate the persisting benefits of a popular physics-based gradient loss term [19] with increasing model size.
We provide an overview of related efforts in Section 2. In Section 3, we provide information on the BLASTNet 2.0 and Momentum128 3D SR datasets. Our benchmark setup is described in Section 4, with results discussed in Section 5, before the conclusions in Section 6.
Figure 1: Summary of this study.
Flow Physics Simulation Datasets Numerical simulations accurately describe flow physics, as long as the simulation grid resolves the smallest lengthscales associated with turbulent dissipation [26]. With up to  $\mathcal{O}(10^{9})$  voxels,  $\mathcal{O}(10^{6})$  core-hours of simulation time, and  $\mathcal{O}(10^{4})$  cores on parallel computing facilities [27-30], high-fidelity DNS of many real-world flows cannot be performed
due to prohibitive costs. Thus, it is common to employ coarser grids with large-eddy simulations (LES) [15, 31], or by only evolving time-/ensemble-averaged quantities with Reynolds-averaged Navier-Stokes (RANS) simulations [32, 33] – both of which rely on turbulence models that can be discovered from DNS data. DNS data is also useful for applications involving scientific imaging [34], spatio-temporal modeling [35], and solving inverse problems [18]. As shown in Table 1, many existing flow simulation datasets focus on LES and RANS simulations due to data storage constraints. McConkey et al. [32] released a dataset for improving turbulence models in incompressible non-reacting RANS. AirfRANS [33] provides both 2D incompressible and compressible non-reacting RANS data, specifically on airfoil configurations. For reacting flows, Huang [31] released a 2D LES dataset for developing reduced-order models. The largest flow physics dataset, the Johns Hopkins Turbulence Database (JHTDB) [36], provides 3D DNS data from turbulent incompressible non-reacting flow simulations. Since these datasets are either 2D, incompressible, or non-reacting, they are not suitable for many applications involving aerodynamics, propulsion, or chemical processes. This is one reason why ML studies involving these applications employ self-generated inaccessible datasets [37, 38] – introducing challenges to open model evaluation. To address these gaps, we curate BLASTNet 2.0, a 3D turbulent compressible reacting and non-reacting flow dataset, which targets a balance of diversity, fidelity, and size.
Table 1: Comparison of BLASTNet 2.0 (in bold) with selected flow simulation datasets.
Datasets	BLASTNet	JHTDB [36]	McConkey et al. [32]	AirfRANs [33]	Huang [31]
2.0	1.0 [9]
Size [TB]	2.2	0.225	490	0.014	0.066	0.209
Configs.	34	10	10	29	1,000	1
Full-domain Samples	744	120	15,791	29	1,000	30,000
Fidelity	DNS	DNS	RANS	RANS	LES
Spatial Dimensions	3D	3D	2D,3D	2D	2D
Primitive Variables1	ui,p,T,ρ,Yk	ui,p,ρ,bi	<ui>,<p>	<u>i,<p>,<T>,<ρ>	<widetildeT,ρ,Yk
Number of PDEs2	5 to 27	4 to 7	3 to 4	4 to 5	7
Compressible Flow?	Yes	No	No	Yes3	Yes
Multi-physics?	Reactions4	MHD5	No	No	Reactions
Host	Kaggle	SciServer [6]	Kaggle	Sorbonne	UMich
Format	fp32 binaries	.h5	.npy	.vtu,.vtp	ascii
SR for Turbulent Flows Within experimental flow measurements, deep learning-based SR approaches have been employed towards improving Schlieren [40], particle-image-velocimetry [41], and tomography [42] techniques, which can often be limited by resolution restrictions. In many practical engineering analyses, coarse-grained simulations with under-resolved grid sizes are often used to bypass the extensive costs of fully-resolved DNS. A dominant source of error from this approach involves missing physics that arise from the under-resolved grid, as will be detailed in Section 4.2. While algebraic turbulence models have been traditionally used to represent the under-resolved physics [43, 44], specific algebraic models are effective only in specific flow configurations [45]. SR and related upsampling approaches have been proposed as a versatile alternative for correcting under-resolved information from coarse-grid simulations between numerical time-stepping [37, 46, 47], which would require considerations of real-time inferencing. Studies on turbulent SR have focused on demonstrating feasibility [37, 41, 46–48], mostly by modifying existing image SR models. Due to memory constraints, many of these studies focus on 2D configurations [41, 46, 48], with 3D SR investigations only demonstrated recently [37, 47]. As such, there has not yet been a detailed and reproducible benchmark study comparing SR models for 3D turbulent flows [38, 49].
$^1$ Filtered and time-/ensemble-averaged quantity  $\phi$  is expressed with  $\overline{\phi}$  and  $\langle \phi \rangle$ , respectively. In compressible LES, Favre-filtering  $\widetilde{\phi}$  [39] is typically used.  $T$  refers to temperature; more definitions in Equation (1).
$^{2}$ PDEs for continuity, energy, momentum, chemical species, and magnetic field  $b_{i}$ .
This dataset contains a blend of compressible and incompressible flow samples.
Majority are reacting flows. Some non-reacting flows are present.
Only one of the ten configurations considers magneto-hydrodynamics (MHD).
Scientific ML Approaches Scientific ML approaches can involve developing custom architectures with implicit biases that suit specific problems and modifying loss functions with constraints related to governing equations [50]. Firstly, model architectures such as NUNet [51], MeshGraphNet [52], and Fourier Neural Operators (FNO) [53] employ graph and spectral convolution layers to ensure that flow predictions are mesh invariant. While FNOs have been successfully employed towards canonical laminar flow configurations, previous studies have demonstrated that these models are insufficiently expressive for complex configurations, involving turbulent [35] and multi-physics [25] flows. This has been attributed to regularization properties of spectral convolution layers by the FNO's original developers [25]. Recently, convolution FNO (Conv-FNO) [25] models have been proposed to ameliorate these underfitting issues by embedding convolutional blocks within FNO blocks. Secondly, modifying the loss functions can involve adding a regularization term based on the residuals of the entire governing equations [54]. A softer approach involves regularizing with individual operators within the governing equation (such as continuity, advection, diffusion, and source terms) [37]. In this work, we benchmark models based on both loss function and architecture approaches, i.e., a model with a gradient-based loss function [19] (which biases ML optimization towards turbulence applications; see Appendix F.3) and a Conv-FNO model, respectively.
SR Benchmarks SR via deep learning has been subjected to numerous competition and benchmark studies that target various aspects of 2D image SR, including 2K-images [55], night photography [56], spectral recovery [57], and satellite images [58]. For 3D SR, benchmarks for video [59], medical resonance imaging [60], and 3D microscopy applications [61] have been performed. Another method for studying the behavior of deep learning models involves the construction of empirical scaling relationships [62]. This type of analysis is useful for studying resource requirements of large language models, and was briefly been employed for studying 2D SR with a U-Net model [63]. Given the potential real-time computing applications of turbulent SR, this analysis provides useful information on the relationship between model size, cost, and predictive performance.
3.1 BLASTNet 2.0
BLASTNet 2.0 consists of turbulent compressible flow DNS data, on Cartesian spatial grids, generated by solving governing equations for mass, momentum, energy, and chemical species, respectively:
$$
\partial_ {t} \rho + \nabla \cdot (\rho \mathbf {u}) = 0, \tag {1a}
$$
$$
\partial_ {t} (\rho \mathbf {u}) + \nabla \cdot (\rho \mathbf {u} \otimes \mathbf {u}) = - \nabla p + \nabla \cdot \boldsymbol {\tau}, \tag {1b}
$$
$$
\partial_ {t} \left(\rho e ^ {t}\right) + \nabla \cdot \left[ \mathbf {u} \left(\rho e ^ {t} + p\right) \right] = - \nabla \cdot \mathbf {q} + \nabla \cdot \left[ (\boldsymbol {\tau}) \cdot \mathbf {u} \right], \tag {1c}
$$
$$
\partial_ {t} \left(\rho Y _ {k}\right) + \nabla \cdot \left(\rho \mathbf {u} Y _ {k}\right) = - \nabla \cdot \mathbf {j} _ {k} + \dot {\omega} _ {k}, \tag {1d}
$$
with density  $\rho$ , velocity vector  $\mathbf{u}$ , pressure  $p$ , specific total energy  $e^t$ , stress tensor  $\tau$ , and heat flux  $\mathbf{q}$ .  $Y_{k}$ ,  $\mathbf{j}_{k}$ , and  $\dot{\omega}_k$  are the mass fraction, diffusion flux, and source term for chemical species  $k = [1,N_s - 1]$ , where  $N_{s}$  is the number of species. Molecular fluxes are typically modeled using the mixture-averaged diffusion model.
The BLASTNet 2.0 dataset is developed with these properties in mind:
Fidelity All DNS data is collected from well-established numerical solvers [29, 64-67] with spatial discretization schemes ranging from 2nd- to 8th-order accuracy, while time-advancement accuracy range from 2nd- to 4th-order. Low-order schemes require finer discretizations compared to high-order schemes, to achieve similar accuracy and numerical stability [68]. However, all simulations are spatially resolved to the order of the Kolmogorov lengthscale, ranging from 3.9 to  $41\mu \mathrm{m}$  depending on the configuration, with a corresponding temporal discretization that ensures numerical stability.
Size and Diversity BLASTNet 2.0 contains a total of 744 full-domain samples (2.2 TB) from a diverse collection of 34 simulation configurations: non-reacting decaying homogeneous isotropic turbulence (HIT) [20], reacting forced HIT [67], two parametric variations of reacting jet flows [29], six configurations of non-reacting transcritical channel flows [69], a reacting channel flow [28], a partially-premixed slot burner configuration [27], and 22 parametric variations (with different turbulent and chemical timescales) of a freely-propagating flame configuration [70].
Community-involvement BLASTNet 2.0 consists of data contributions from six different institutions. As mentioned in Appendix C, our long-term vision and maintenance plan for this dataset involves seeking additional contributions from members of the broader flow community.
Cost-effective Storage, Distribution, and Browsing To circumvent Kaggle storage constraints, we partition the data into a network of  $< 100$  GB subsets, with each subset containing a separate simulation configuration. This partitioned data can then be uploaded as separate datasets on Kaggle. To consolidate access to this data, all Kaggle download links are presented in https://blastnet.github.io, with the inclusion of a bash script for downloading all data through the Kaggle API. In addition, Kaggle notebooks are attached to each subset to enable convenient data browsing on Kaggle's cloud computing platform. This approach enables cost-effective distribution of scientific data that adheres to FAIR principles [71], as further detailed in Appendix C.
Consistent Format Data, generated from different numerical solvers, initially exists in a range of formats (.vtk, .vtu, .tec, and .dat) that are not readily formatted for training ML models. Thus, all flowfield data are processed into a consistent format - little-endian single-precision binaries that can be read with np.fromfile/np.memmap. The choice of this data format enables high I/O speed in loading arrays. We provide .json files that store additional information on configurations, chemical mechanisms and transport properties. See Appendix D for more details.
Licensing and Ethics All data is generated by the present authors and licensed via CC BY-NC-SA 4.0. Other than the contributors' names and institutions, no personal-identifiable information is published in this data. No offensive content is published with this flow physics dataset. Further discussion on negative impact is provided in Section 6.
3.2 Momentum128 3D SR Dataset
BLASTNet 2.0 is further processed for training due to constraints in (i) memory and (ii) grid properties. Currently, the single largest sample (92 GB) in BLASTNet 2.0 contains 1.3B voxels and 15 channels, which cannot fit into typical GPU memory. In addition, the spatial grid is stretched depending on the resolution requirements of the flow domain. As shown in Figure 1, we circumvent these two issues by sampling  $128^{3}$  sub-volumes of density  $\rho$  and velocity  $\mathbf{u}$  from the uniform-grid regions from all BLASTNet data. This results in 12,750 sub-volume samples (427 GB). We choose this sub-volume size to enable  $32\times \mathrm{SR}$  (the resulting feature subvolume is  $4^{3}$  which is larger than a kernel size of 3), while maintaining a low memory footprint. In order to develop a compressible turbulence benchmark dataset that can be easily downloaded, we select 2,000 sub-volumes to form a 67 GB dataset that can fit into a single Kaggle repository. To ensure that these 2,000 samples are representative of the different flows encountered in each configuration, we:
Figure 2: Statistics of the specific kinetic energy  $\rho e^k$  of each  $128^{3}$  subvolume in the Momentum128 3D SR dataset. Marker type represents DNS configuration. See detailed legend in Appendix E.1.5
1. Extract mean, variance, skewness, and kurtosis (statistical moments for characterizing turbulence [26]) from the three velocity components of 12,750 sub-volumes.
2. Apply k-means clustering with the elbow method (using the statistical moments as features) to partition the sub-volumes in 18 clusters.
3. Select 2,000 samples while ensuring that the proportion of clusters are well-balanced.
The resulting sub-volumes form the labels of BLASTNet Momentum128 3D SR dataset. Figure 2 demonstrate the mean and standard deviation of the specific kinetic energy:
$$
\rho e ^ {k} = \rho \left(u _ {1} ^ {2} + u _ {2} ^ {2} + u _ {3} ^ {2}\right) / 2, \tag {2}
$$
which we use to characterize all channel variables. Each distinct marker represents a different simulation configuration. Since flows from the same configuration possess similar statistics, the different configurations from BLASTNet 2.0 can result in a dataset with a variety of flow conditions.
Due to stochastic and chaotic nature of turbulence [72], it is not possible to obtain matching pairs of coarse (also known as implicit LES [45]) and fine DNS data. Thus, we employ a canonical method [37, 45, 73] for obtaining implicit LES surrogates, known as finite-volume optimal LES [74] (see Appendix E.1.6 on their validity). Specifically, we Favre-filter [39] and downsample the labels by 8, 16, and  $32 \times$  to obtain a representative range of coarse resolution samples (LES is typically an order of magnitude coarser than DNS [26, 75]) to generate inputs for turbulent SR:
$$
\widetilde {\phi} = \frac {1}{\bar {\rho} V _ {f}} \int_ {V _ {f}} \rho \phi d V _ {f} \tag {3}
$$
where  $\overline{\phi}$  denotes a uniform-filtered quantity,  $\widetilde{\phi}$  is a Favre-filtered quantity, and  $V_{f}$  is a subvolume with the size of the filter width. In our SR dataset, the channels of each label correspond to  $\phi = \{\rho ,u_1,u_2,u_3\}$ , while the feature channels consist of  $\phi_f = \{\overline{\rho},\widetilde{u}_1,\widetilde{u}_2,\widetilde{u}_3\}$ . For the purpose of the present benchmark study, we further split the 2,000 sub-volumes as follows:
Train, Validation, and Baseline Test Sets 80:10:10 split via random selection with a uniform distribution. The training set contains 1,382 samples, and both validation and baseline test sets contain 173 samples each.
Parametric Variation Set A 144-sample subset for model evaluation from an unseen parametric variation configuration with approximately  $15\%$  higher mean velocities and velocity fluctuations than the train, validation, and baseline test sets.
Forced HIT Set A 128-sample subset for model evaluation from an unseen flow type (forced HIT) with 30-fold higher pressure and 34-fold lower velocity fluctuations.
4 Benchmark Configuration
4.1 Models and Methods
As shown in Figure 1, three well-studied 2D ResNet-based [76] SR models are modified from their original repositories for 3D SR: (i) Residual-in-Residual Dense Block (RRDB) [22], (ii) Enhanced Deep Residual Super-resolution (EDSR) [23], and (iii) Residual Channel Attention Networks (RCAN) [24]. Convolution networks possess inductive biases that are suitable for problems involving spatial grids such as in flow physics [50, 77]. We choose to study these models due to their differences in architecture paradigms. Specifically, RRDB employs residual layers within residual layers; EDSR features an expanded network width; RCAN utilizes long skip connections and channel attention mechanisms. In addition, we consider two additional scientific ML approaches: (i) a Conv-FNO model [25], modified for SR (see details in Appendix F.2), and (ii) an RRDB model regularized with the weighted MSE of the gradient of channel variables  $\lambda \Delta^2 \sum_{k=1}^{3} \mathrm{MSE}[(\nabla \hat{\phi})_k, (\nabla \phi)_k]$  [19] to the loss  $(1 - \lambda) \mathrm{MSE}(\hat{\phi}, \phi)$ , where  $\Delta$  is the distance between each voxel. Details on the gradient-based loss and its weighting factor  $\lambda$  are provided in Appendix F.3. We compare model predictions with a baseline approach, i.e., tricubic interpolation. To investigate the scaling behavior of the model architectures, we vary the number of parameters by changing the network depth and width.
Similar to other turbulent SR studies [37, 46], all models are trained with mean-squared-error (MSE) loss, unless otherwise stated. For evaluation, we select models with the best MSE after training for 1,500 epochs with a batch size of 64 across 16 Nvidia V100 GPUs. Learning rate is initialized at  $1\mathrm{e} - 4$  and halved every 300 epochs. Both the number of training iterations and learning scheduling are chosen to match other SR studies [22-24] and are found to be sufficient for the SR predictions, as will be shown in Section 5. All other hyperparameters are maintained from their original studies, with He initialization [78] used on all initial model weights. Data augmentation is performed via variants of random rotation and flip – modified to ensure that augmented data remains consistent with continuity (Equation (1a)). Training is performed with automatic mixed-precision from Lightning 1.6.5 [79]. Prior to training, data is normalized with means and standard deviations of density and velocity extracted from the train set. During evaluation, this normalization resulted in poor accuracy for the Forced HIT set, due to the significantly different magnitudes of density and velocity. However, Section 5 will show that good performance can be achieved when normalization is performed with the mean and standard deviation of each distinct evaluation set. Thus, all evaluation sets are normalized
with their own mean and standard deviation, prior to testing. All 40 model variations are trained with three different seeds, resulting in a total computational cost of approximately 15,000 GPU-hours on the Lassen Supercomputer [80]. Further information on model hyperparameters, data augmentation, training, normalization, as well as links to code and model weights are found in Appendix A, F, and G.
We compare the performance of each model by examining local and global quantities of each sample. For the local quantities, we employ Metrics = {SSIM, NRMSE}, where SSIM is the 3D extension [81] of the structural similarity image measure [82] and NRMSE is the normalized root-mean-squared error (see Appendix F.1). For quantities with multiple channels: