SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gaussian noise is applied on iterations that the pushforward trick is unused. Several papers claim that adding noise is a simple method to help the model become more robust to distribution shift during longer rollouts [58, 71, 72]. Similar to F-FNO, when used, we add noise with zero mean and standard deviation of 0.01. As Gaussian noise is spread uniformly across frequencies, it may also help the model account for high frequencies [28].
Pushforward trick. Algorithm 1 shows sample pseudo-code for the pushforward trick [57]. The model is trained by performing multiple inferences for successive timesteps. Similar to using Gaussian noise, this trick helps the model be robust to distribution shifts during rollouts. The advantage is that the errors introduced during the pushforward steps better match the distribution of errors that accumulate during rollouts. Thus, we are essentially training the model to correct for its own distribution shift during rollout. We use a ramp-up when applying the pushforward trick: the percentage of training iterations completed is used as the likelihood that we perform pushforward steps. Initially, there is a  $0\%$  chance of pushing forward. After  $50\%$  of training iterations, there is a  $50\%$  chance of applying the pushforward trick. We found this to be necessary and that applying the pushforward trick on every iteration would fail.
Algorithm 1 Pushforward Trick.
Require: input is the model's input;  $N$  is the number of pushforward steps 1: for  $n\in \{0,\dots ,N\}$  do 2: input  $\leftarrow$  model(input) Step through  $N$  timesteps 3: end for 4: input  $\leftarrow$  model(input) Track gradients for only the final forward pass.
Temporal bundling is used so that the model outputs the solution for multiple future timesteps with one inference [57]. By having the model output the solution for multiple future timesteps, we can reduce the number of times the model is used. Since some error is introduced by each inference during rollout, temporal bundling should reduce the rate that error is accumulated. This is again beneficial for maintaining accuracy and stability during long rollouts. With a temporal bundle of size  $k$ , the forward propagator outputs a vector  $\mathcal{F}(u([t_{n - k}, t_{n - 1}])) = u([t_n, t_{n + k - 1}])$ .
Algorithm 2 Main Training Loop, with Pushforward Trick.
Require: data, the set of simulations;  $N$ , the number of pushforward steps  
1: sample  $t$  from  $\{t_k, t_{max}\}$   
2: input  $\leftarrow$  data  $[t - k\Delta t, t - \Delta t]$   
3: if  $N = 0$  then  
4: input  $\leftarrow$  input +  $\mathcal{N}(0, 0.01)$   
5: end if  
6: output = PushForwardTrick(input, N)  
7: target  $\leftarrow$  data  $[t + Nk, t + (N + 1)k - 1]$   
8: loss(output, target)
Table 7: Generic hyperparameter settings for the temperature experiments.
Hyperparameter	Value
Number of Epochs	250
Batch Size	8
Optimizer	AdamW
Weight Decay	0.01
Base LR	1e-3 (or 5e-4)
LR Warmup	Linear, 3%
LR Scheduler	Step
Step Factor	0.5
Step Patience	75 Epochs
History Size	5
Future Size	5
Table 8: Generic hyperparameter settings for the flow experiments.
Hyperparameter	Value
Number of Epochs	25
Batch Size	16
Optimizer	AdamW
Weight Decay	1e-4
Base LR	5e-4
LR Warmup	Linear, 3%
LR Scheduler	Cosine Annealing
Min LR	1e-6
History Size	5
Future Size	5
C.3 Hyperparameter Settings
We mostly use generic hyperparameter settings that we found work well across all models. Each individual model, however, has its own specific parameters, such as hidden channels, Fourier modes, etc. These parameters are tuned on the subcooled Pool Boiling dataset and reused for the other datasets. The settings for the temperature and flow experiments can be found in Tables 7 and 8. All models are trained using AdamW [73]. For each temperature model, we use  $3\%$  of the total iterations for learning rate warmup to reach the base learning rate. For the remaining iterations, we apply a learning rate decay. For  $\mathrm{UNet}_{\mathrm{mod}}$ , we reduce the base learning rate to 5e-4 in order to avoid instability we experienced during some training runs. We also perform gradient clipping to clip the gradient  $l_{2}$ -norm to be at most one. For data augmentation, we perform horizontal reflections only for pool boiling.
The Fourier models that are resolution invariant are trained at half resolution and evaluated at full resolution. We found that training at the full resolution gave similar results, but took much longer or required distributed training. The Fourier models all take the  $xy$ -coordinate grid as input [69]. So, the input is  $(x,y,T([t - k,t - 1]),v([t - k,t - 1]))$ , where  $T$  is the temperature map and  $v$  is the velocity map. Each Fourier model uses the 64 lowest frequency modes, except UNO, which uses the 32 lowest frequencies in the bottom of the "U" where the resolution is too small to use 64 modes. As the domain is non-periodic, we do Fourier continuation by padding the domain with zeros [13]. FNO is tuned with varying number of hidden channels. We achieve the best results with four layers and 64 hidden channels per layer. For G-FNO, we use the same settings, but halve the number of hidden channels to account for the increase in size of the weight tensor.
C.4 Learning Temperature Dynamics Results
We show the complete listing of results for predicting temperature dynamics in Tables 9, 10, 11, 12, and 13. As a general trend, we see that  $\mathrm{UNet}_{\mathrm{bench}}$  consistently performs the best. In some metrics,  $\mathrm{UNet}_{\mathrm{mod}}$  closely matches or is better than  $\mathrm{UNet}_{\mathrm{bench}}$ , but it tends to have higher errors at bubble interfaces and domain boundaries. The FNO variants also perform well, but slightly worse than
the UNet architectures. The better performance of UNet architecture might stem from the nature of convolutions to function as effective edge detectors. Thus, the UNet variants can handle sharp interfaces between liquid and vapor. Since the Fourier layers implicitly assume periodicity, they tend to act as "smoothers". Qualitatively, they appear to soften the sharp interfaces, resulting in errors along the region surrounding the bubbles. Furthermore, it's important to note that in FNO, frequency truncation primarily preserves the low-frequency modes. While this is well-suited for modeling and capturing general patterns and large-scale dynamics, it may not adequately represent the small-scale structures present around bubble interfaces, the heater surface, and condensation trails. There is existing work that makes similar observations [55, 56]. In our efforts to train FNO for the solution of complex PDEs with high-frequency information, we have experimented with increasing the number of effective Fourier modes to 2/3 of the training data resolution [74]. These results also underscore the importance of regularization in low-data regimes. Among the FNO variants, those that effectively reduce the number of parameters while retaining model expressiveness (T-FNO and F-FNO) outperformed their larger, dense counterparts (FNO, UNO, and G-FNO). Simply reducing the number of parameters in these three models did not lead to substantially better results. This observation highlights the potential challenges these models may encounter when applied in contexts where the training dataset is relatively limited, as is the case for this problem.
Table 9: Temperature Prediction: Pool Boiling Subcooled.
UNetbench	UNetmod	T-FNO	F-FNO	G-FNO	FNO	UNO
Rel. Err.	0.036	0.051	0.052	0.050	0.064	0.070	0.062
RMSE	0.035	0.049	0.050	0.048	0.062	0.068	0.059
BRMSE	0.073	0.146	0.118	0.124	0.142	0.149	0.145
IRMSE	0.113	0.157	0.189	0.155	0.236	0.269	0.220
Max Err.	2.1	1.937	2.32	2.21	2.825	2.806	2.659
F. Low	0.281	0.348	0.373	0.472	0.507	0.625	0.440
F. Mid	0.266	0.386	0.390	0.375	0.519	0.559	0.453
F. High	0.040	0.059	0.055	0.057	0.061	0.063	0.068
Table 10: Temperature Prediction: Pool Boiling Saturated.
UNetbench	UNetmod	T-FNO	F-FNO	G-FNO	FNO	UNO
Rel. Err.	0.035	0.039	0.052	0.053	0.066	0.078	0.072
RMSE	0.035	0.039	0.052	0.052	0.065	0.076	0.071
BRMSE	0.082	0.147	0.165	0.163	0.218	0.214	0.214
IRMSE	0.052	0.083	0.125	0.095	0.152	0.140	0.116
Max Err.	1.701	1.785	2.055	1.788	1.793	2.106	2.23
F. Low	0.257	0.241	0.373	0.416	0.463	0.633	0.568
F. Mid	0.268	0.273	0.377	0.399	0.496	0.606	0.532
F. High	0.023	0.042	0.053	0.052	0.070	0.062	0.069
Table 11: Temperature Prediction: Pool Boiling Gravity.
UNetbench	UNetmod	T-FNO	F-FNO	G-FNO	FNO	UNO
Rel. Err.	0.042	0.051	0.062	0.061	0.103	0.104	0.081
RMSE	0.040	0.049	0.059	0.058	0.098	0.099	0.077
BRMSE	0.070	0.139	0.150	0.131	0.188	0.209	0.184
IRMSE	0.108	0.201	0.266	0.239	0.412	0.457	0.335
Max Err.	2.92	2.846	3.626	3.22	3.307	3.946	3.667
F. Low	0.491	0.466	0.500	0.578	1.035	1.103	0.746
F. Mid	0.275	0.326	0.440	0.423	0.704	0.724	0.548
F. High	0.033	0.049	0.054	0.049	0.061	0.067	0.077
C.5 Learning Flow Dynamics Results
Table 14 shows the predicted temperature,  $x$  velocity, and  $y$  velocity during rollout of 200 timesteps on the pool boiling subcooled dataset. We observe an interesting phenomenon: UNetbench goes from
Table 12: Temperature Prediction: Flow Boiling Gravity.
UNetbench	UNetmod	T-FNO	UNO
Rel. Err.	0.055	0.095	0.134	0.157
RMSE	0.051	0.088	0.123	0.144
BRMSE	0.080	0.240	0.267	0.301
IRMSE	0.091	0.214	0.351	0.379
Max Err.	2.560	2.800	3.951	3.990
F. Low	0.263	0.388	0.591	0.602
F. Mid	0.281	0.419	0.551	0.692
F. High	0.149	0.276	0.389	0.449
Table 13: Temperature Prediction: Flow Boiling Inlet Velocity. All models have substantially higher error compared to the other datasets. This is likely caused by the higher velocity. Since the temporal discretization is quite coarse, each bubble will take a large "jump" between frames that is difficult to capture accurately.
UNetbench	UNetmod	T-FNO	UNO
Rel. Err.	0.106	0.122	0.202	0.225
RMSE	0.097	0.111	0.184	0.205
BRMSE	0.271	0.315	0.486	0.471
IRMSE	0.178	0.193	0.252	0.319
Max Err.	2.900	2.862	3.527	3.992
F. Low	0.571	0.584	1.061	0.981
F. Mid	0.511	0.566	1.084	1.334
F. High	0.264	0.315	0.494	0.523
being the best model at predicting temperature, to being among the worst. This is the simplest model, so it is understandable that its limited capacity prevents it from learning more complex functions. In comparison,  $\mathrm{UNet}_{\mathrm{mod}}$  performs incredibly well. T-FNO and F-FNO also continue to very well. The best performing model is P-UNet<sub>mod</sub>, which was trained using the pushforward trick. It achieves the best score in all metrics, except, surprisingly, for the maximum error. UNO experiences some divergence. Initially, it makes good predictions (perhaps better than the UNet architectures), but over long rollouts, its predictions tend toward infinity. Since UNO is the model that experiences the most striking divergence, we compare its 1D radially averaged power spectrum with  $\mathrm{UNet}_{\mathrm{mod}}$  and P-UNet<sub>mod</sub> in Figure 10. We are able to reproduce findings that show that autoregressive Fourier models may experience aliasing errors, causing them to diverge [56].
Figure 10: 1D Radially Averaged Power Spectrum of the simulation temperature and model rollout temperature. We see that initially, all of the models follow the simulations frequency fairly well, but UNO quickly diverges. This phenomena has been reported [56] as being aliasing errors. The plots are on log-scale, so curves below the simulation's are more accurate than curves above the simulation.
Table 14: Coupled Velocity and Temperature Prediction: Pool Boiling Subcooled<sub>0,1</sub>. P-UNet<sub>mod</sub> was trained using the pushforward trick. It gets the best results during rollout. Notably, F-FNO performs better than UNet<sub>bench</sub> and UNet<sub>mod</sub>. This is a sharp contrast to the temperature predictions, where the UNet variants were far better. UNO diverged during rollout. This is likely because is mostly uses low frequencies. The error in the high frequencies accumulates during rollout, and it diverges.
Metric	UNetbench	UNetmod	P-UNetmod	T-FNO	UNO	F-FNO
Temp.	Rel. Err.	0.108	0.074	0.040	0.066	0.322	0.067
RMSE	0.105	0.072	0.039	0.064	0.051	0.065
BRMSE	0.248	0.211	0.113	0.142	0.050	0.237
IRMSE	0.178	0.191	0.129	0.212	0.077	0.201
Max Err.	2.83	2.340	2.741	2.566	7.492	2.488
F. Low	1.407	0.495	0.244	0.531	4.902	0.498
F. Mid	0.740	0.583	0.280	0.518	1.594	0.507
F. High	0.090	0.076	0.051	0.061	0.442	0.083
x Vel.	Rel. Err.	0.681	0.553	0.429	0.642	1.834	0.547
RMSE	0.021	0.015	0.012	0.018	0.051	0.015
BRMSE	0.023	0.023	0.022	0.023	0.050	0.024
IRMSE	0.051	0.049	0.043	0.049	0.077	0.050
Max Err.	0.294	0.320	0.353	0.32	0.574	0.334
F. Low	0.329	0.225	0.142	0.258	0.677	0.189
F. Mid	0.115	0.105	0.079	0.110	0.328	0.107
F. High	0.012	0.012	0.011	0.012	0.092	0.012
y Vel.	Rel. Err.	0.563	0.400	0.311	0.415	1.170	0.397
RMSE	0.021	0.015	0.012	0.016	0.044	0.015
BRMSE	0.015	0.012	0.010	0.011	0.035	0.013
IRMSE	0.046	0.046	0.038	0.045	0.066	0.047
Max Err.	0.292	0.272	0.280	0.290	0.288	0.289
F. Low	0.387	0.190	0.132	0.180	0.660	0.176
F. Mid	0.128	0.112	0.085	0.111	0.240	0.106
F. High	0.011	0.010	0.009	0.010	0.067	0.010
D Physical and non-dimensional values
D.1 Fluid properties
Table 15 provides a comprehensive list of physical properties relevant to multiphase flow. These properties are presented with their respective symbols, units, and descriptions. It is important to consider the properties of the liquid and vapor phases separately in multiphase flows, as they exhibit distinct characteristics due to the phase change phenomenon. Additionally, the temperatures of the heater and liquid are also taken into account, as they significantly influence bubble dynamics and flow behaviors. The saturation temperature, which marks the initiation of bubble generation, is determined by the specific fluid being studied.
Table 15: Physical properties with symbol and unit
Symbol	Unit	Description
ρl	kg/m3	Liquid density
ρv	kg/m3	Vapor density
μl	N·s/m2	Dynamic viscosity of liquid
μv	N·s/m2	Dynamic viscosity of vapor
Cp,l	J/kg·K	Specific heat capacity of liquid
Cp,v	J/kg·K	Specific heat capacity of vapor
kl	W/m·K	Thermal conductivity of liquid
kv	W/m·K	Thermal conductivity of vapor
hvl	J/kg	Latent heat
g	m/s2	Gravitational acceleration
σ	N/m	Surface tension
Twall	K	Heater temperature
Tbulk	K	Bulk temperature
Tsat	K	Saturation temperature
D.2 Conversion to Non-dimensional parameters
Table 16 provides a comprehensive list of non-dimensional variables used in this study, including their symbols, definitions, and descriptions. For example, the non-dimensional time is obtained by dividing the characteristic length scale by the characteristic velocity scale. These non-dimensional variables are necessary to solve the non-dimensionalized governing equations such as the continuity, momentum, and energy equations. Additionally, non-dimensional properties such as density, dynamic viscosity, specific heat capacity, and thermal conductivity are also considered in the context of multiphase flow. Notably, representative parameters in fluid mechanics, thermodynamics, and heat transfer, such as the Reynolds number  $(Re)$ , Froude number  $(Fr)$ , Prandtl number  $(Pr)$ , Stefan number  $(St)$ , Weber number  $(We)$ , and Peclet number  $(Pe)$ , are used to obtain simulation results. By employing non-dimensionalization, the effects of different physical quantities can be studied independently of their specific units, facilitating a deeper understanding of the underlying phenomena.
E Simulation Details
E.1 Multiphase Simulations
Numerical simulations of multiphase flows with phase changes have been studied using various techniques to model the behavior at the liquid-vapor interface and track its evolution over time. Two commonly used methods for handling boundary conditions related to surface tension and evaporation are the ghost fluid method (GFM) and the continuum surface force method (CSF). The GFM enforces a sharp jump in pressure, velocity, and temperature across the interface, while the CSF diffuses the forcing within the vicinity of the interface for a smoother transition. The choice between these approaches involves a trade-off between accuracy and stability, with GFM offering higher accuracy but lower stability compared to CSF. Interface tracking is typically achieved implicitly using level-set or volume of fluid (VOF) techniques.
Table 16: Formulae used for conversion of real world values to non-dimensional values
Symbol	Definition	Description
ρ*	ρv/ρl	Non-dimensional density
μ*	μv/μl	Non-dimensional viscosity
Cp*	Cp,v/Cp,l	Non-dimensional specific heat capacity
k*	kv/kl	Non-dimensional thermal conductivity
l0	√σ/gΔρ	Characteristic length scale
u0	√gl0	Characteristic velocity scale
t0	l0/u0	Characteristic time scale
T*wall	(Twall-Tbulk)/(Twall-Tbulk)=1	Non-dimensional heater temperature
T*bulk	(Tbulk-Tbulk)/(Twall-Tbulk)=0	Non-dimensional bulk temperature
T*sat	(Tsat-Tbulk)/(Twall-Tbulk)	Non-dimensional saturated temperature
Re	ρlu0l0/μl	Reynolds number
Fr	u0/√gl0	Froude number
Pr	μlCp,l/kl	Prandtl number
St	Cp,l(Twall-Tbulk)/hvl	Stefan number
We	ρlu02l0/σ	Weber number
Pe	Re·Pr	Peclet number
Researchers have employed these methods to study and model various aspects of multiphase flows with phase changes. For example, Gibou et al. [75] used a level-set method with sharp interfacial jump conditions within the framework of the GFM to model homogeneous two-dimensional evaporation and film boiling. Son and Dhir [76, 77] extended this approach to perform heterogeneous pool boiling calculations involving single and multiple bubbles. Majority of their initial work focused on model development and verification using two-dimensional (2D) simulations, since real world three-dimensional (3D) calculations were expensive due to limitations of the software framework.
Efforts have also been made to perform high-fidelity 3D simulations of pool boiling by combining different techniques. Yazadani et al. [78] conducted critical heat flux (CHF) calculations on earth gravity using a combination of VOF and CSF methods. Sato et al. [79, 11], on the other hand used the level-set method in combination with CSF for their simulations. These studies highlighted the computational cost associated with boiling simulations which had to be mitigated by performing low resolution calculations. More recently, Dhruv et al. [31, 32] used a combination of level-set and GFM methods for gravity scaling analysis of pool boiling at finer resolution than previous studies using adaptive mesh refinement (AMR) within the framework of FLASH. These simulations were applied to study effects of gravity on boiling heat transfer which lead to verification of experiment based heat flux models and enabled the quantification of turbulent heat flux associated with bubble dynamics during bubbly and slug flow [32]. The implementation of multiphase models within FLASH has transitioned to Flash-X, which leverages state-of-art AMR techniques and heterogeneous supercomputing architectures to significantly improve performance of boiling calculations. The BubbleML dataset is curated from simulations carried out with the Flash-X framework. Figures 11 and 12 illustrate the boundary conditions and describe the parameters of the simulations comprising the dataset.
An important note is that simulations still heavily rely on experimental observations to determine input conditions such as nucleation site distribution, bubble nucleation frequency, and solid-liquid-vapor contact angle dynamics [31]. As a result, simulations serve as an effective tool to understand and quantify trends in boiling regimes, rather than attempting to replicate experiments precisely. This opens up an opportunity for the integration of data-driven ML techniques, which can leverage diverse datasets to make informed predictions for boiling phenomena.
E.2 Fluid parameters
The input configuration file of a simulation requires the inclusion of certain parameters, which remain constant for a specific fluid. For our simulations using FC-72 (Perfluorohexane,  $C_6F_{14}$ ), the non-dimensional values for this fluid are provided in Table 17. It is important to note that parameters corresponding to any specific real-world fluid must be converted to these non-dimensional values
Figure 11: Parameters for 2D pool boiling simulations
Figure 12: Parameters for 2D flow boiling simulations
before being input into the simulation configuration files. This conversion allows for consistent and standardized representation of fluid properties in the simulation, enabling accurate and meaningful results to be obtained.
Table 17: Non-dimensional constants for FC-72, the fluid used in BubbleML
Parameter	Variable Name	Non-dimensional Value
Inverse Reynolds Number, 1/Re	ins_invReynolds	0.0042
Non-dimensional density, ρ*	mph_rhoGas	0.0083
Non-dimensional viscosity, μ*	mph_muGas	1.0
Non-dimensional thermal conductivity, k*	mph_thcoGas	0.25
Non-dimensional specific heat capacity, Cp*	mph_CpGas	0.83
Inverse Weber Number, 1/We	mph_invWeber	1.0
Prandtl Number, Pr	ht_Prandtl	8.4