SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Transformer [29] has been proven successful in the NLP field. The design of the multi-head attention (MHA) layer is based on the attention mechanism with Query-Key-Value (QKV). Given the packed matrix representations of queries  $Q$ , keys  $K$ , and values  $V$ , the scaled dot-product attention used by Transformer is given by:
$$
\operatorname {A T T E N T I O N} (\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {V}) = \operatorname {s o f t m a x} \left(\frac {\boldsymbol {Q} \boldsymbol {K} ^ {T}}{\sqrt {D _ {k}}}\right) \boldsymbol {V}, \tag {20}
$$
where  $D_{k}$  represents the dimensions of queries and keys.
The multi-head attention applies  $H$  heads of attention, allowing a model to attend to different types of information.
$$
\operatorname {M H A} (\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {V}) = \operatorname {C O N C A T} \left(\operatorname {h e a d} _ {1}, \dots , \operatorname {h e a d} _ {H}\right) \boldsymbol {W}
$$
$$
\text {w h e r e} \quad \operatorname {h e a d} _ {i} = \operatorname {A T T E N T I O N} \left(\boldsymbol {Q} \boldsymbol {W} _ {i} ^ {Q}, \boldsymbol {K} \boldsymbol {W} _ {i} ^ {K}, \boldsymbol {V} \boldsymbol {W} _ {i} ^ {V}\right), i = 1, \dots , H. \tag {21}
$$
A.8 Additional details for experimental setups
We described the details of the experiments of PbGMR-GMUS and attention-based conditional flow model. We provide the hyperparameters used in the experiments in Table 6.
Cylinder	Sonic	Vascular	Stochastic BFS
PbGMR-GMUS Optimization
Learning rate	1 × 10-4	1 × 10-5	1 × 10-4	1 × 10-4
Optimizer	Adam [51]
Batch size	1
Number of epochs	1500	20000	700	600
MHA/Flow model Optimization
Learning rate	1 × 10-4	1 × 10-5	1 × 10-5	1 × 10-4
Optimizer	Adam
Batch size	5	50	10	5
Number of epochs	90000	240000	240000	220000
Weight decay	1 × 10-5	N/A
PbGMR-GMUS Architecture
Layers of message-passing	3
Hidden dimension	128
Output dimension	256	256	400	256
Activation function	relu
MHA Architecture
Layers of Encoding MHA	2
Layers of Decoding Masked-MHA	1
Hidden dimension	1024	1024	1600	1024
Activation function	gelu [52]
Number of heads	4	8	4	4
Flow model Architecture
Conditioning length	1024
Hidden size	1024
Number of coupling layers	2
Table 6: Hyperparameters
Figure 10: Reconstructor error RMSE with different numbers of centers for cylinder flow, with the training epoch  $= 300$ .
A.9 Performance impact of substituting Transformer with LSTM
In this research, we conduct an ablation study in which we replace the proposed attention-based sequence model in our framework with a Long Short-Term Memory (LSTM) [53] architecture. The aim is to evaluate the impact of the underlying sequence modeling on the performance of our system. The result is listed in Table 7. Following this change, we observe that there is a notable decrease in the performance metrics, suggesting a less optimal fit for the task at hand compared to the Transformer-based model. This outcome underscores the usefulness of attention-based sequence models, like the Transformer, which appears to capture dependencies in the data more effectively than the LSTM. The attention mechanism inherent to Transformers allows the model to focus on different parts of the input sequence dynamically, which may explain the observed performance superiority.
Table 7: The average relative rollout error of two systems, with the unit of  $1 \times 10^{-3}$ .
Dataset-rollout step
Variable	Sonic flow-40	Vascular flow-250
u	v	p	T	u	v	p
PbGMR-GMUS	LSTM	10.3	29.7	1.7	11.1	43.0	46.6
A.10 Performance impact of number of centers
We conduct an ablation study to investigate the effect of the number of centers on the RMSE of our framework's predictions for cylinder flow recovery. The result is in Figure 10. As we progressively increased the number of centers, the recovery RMSE exhibited a decreasing trend, demonstrating an enhancement in prediction accuracy. Interestingly, upon reaching 256 centers, the rate of RMSE decrease started to decelerate significantly. This suggests that the optimal number of centers for cylinder flow is around 256. Beyond this point, further increments do not contribute as much to the improvement in encoding-decoding accuracy, while simultaneously increasing the computational cost. Hence, a count of 256 centers appears to be the sweet spot, offering a good trade-off between recovery accuracy and computational efficiency.
Figure 11: The first row shows meshes and the second row shows the distribution for the centers.
A.11 Centers
The centers distribution for Stochastic BFS can be found in Figure 11.
A.12 Analysis for stochastic BFS
Backward-facing step flow refers to a fluid flow configuration characterized by a sudden contraction followed by an expansion in a channel or pipe. This flow configuration is commonly encountered in various engineering applications, such as in heat exchangers, combustion chambers, and aerodynamic systems. In backward-facing step flow, the fluid initially enters the channel or pipe through an inlet, where it encounters a step or abrupt contraction. As the fluid passes over the step, its velocity increases, and the flow separates, forming a recirculation zone downstream. This recirculation zone is often referred to as the "backflow region." The backward-facing step flow has been extensively studied due to its complex flow characteristics and its relevance in understanding phenomena like separation, reattachment, and flow control.
The simulations and generated samples exhibit remarkable similarity in the contours of both streamwise velocity and wall-normal velocity (Figure 4, 12, 13, 14). This close resemblance demonstrates the high fidelity of the simulation results and the accuracy of the generated samples. The level of detail captured in these contours highlights the effectiveness of the proposed generative framework.
Figure 12: Velocity contour  $(v)$  at time step 40, 80, 120, 160 (left to right) from different samples.
Figure 13: Velocity contour  $(u)$  at time step 40, 80, 120, 160 (left to right) from different LES simulations.
We also randomly choose nine spatial locations to analyze the learning performance, and the coordinates of these points are plotted in Figure 15. Figure 16 and 17 presented the comparison of  $u$ ,  $v$  velocity between the LES simulation, URANS simulation and our model. Generally, the pointwise velocity temporal signals obtained from the generated sample provide a comprehensive representation of the flow characteristics and look visually similar to those obtained from LES simulations. However, the traditional URANS modeling approach cannot capture the intricate fluctuations in the flow field. Note that we also found the generated velocity time signals at points 1 and 2 have a relatively large discrepancy from the LES simulation, and it is explainable. Point 1 is very close to the inlet boundary; therefore, the  $u$  velocity here is very random because of our stochastic velocity inlet condition. Point 2 locates in a region where the BFS flow is not fully developed yet and in this region the  $v$  velocity
Figure 14: Velocity contour  $(v)$  at time step 40, 80, 120, 160 (left to right) from different LES simulations.
Figure 15: Spatial location of the 9 random chosen evaluation points. Point 1:  $(-4.98, 0.016)$ , Point 2:  $(-1.65, 0.55)$ , Point 3:  $(1.68, -0.88)$ , Point 4:  $(0.016, -0.32)$ , Point 5:  $(3.35, 0.22)$ , Point 6:  $(1.68, 0.78)$ , Point 7:  $(5.01, -0.65)$ , Point 8:  $(8.35, -0.11)$ , Point 9:  $(6.68, 0.45)$
Figure 16: Time signal from LES  $(- - - )$ , URANS  $(- - )$ , flow model  $(- - )$  of  $u$  evaluated at 9 points (top to bottom, left to right).
Figure 17: Time signal from LES  $(\text{---})$ , URANS  $(\text{---})$ , flow model  $(\text{---})$  of  $v$  evaluated at 9 points (top to bottom, left to right).
magnitude is very small. These conditions make it hard to capture an accurate distribution of the velocity at these 2 points. Finally, we also compare the velocity distribution between our generated samples and the LES simulation results, as shown in Figure 18, 19. Except for points 1 and 2, the distribution of our generated samples aligns very well with the LES simulations.
Figure 18: Comparison of the distribution of  $u$  velocity between the LES result and our model at 9 evaluation points
Figure 19: Comparison of the distribution of  $v$  velocity between the LES result and our model at 9 evaluation points
A.13 Ablation study
Comprehensive ablation studies are included in this section to demonstrate the effectiveness of each parts. The velocity reconstruction error on the Backward-facing step flow dataset with or without residual connection and centers is shown in Table 8. The result indicates that each component can improve the graph reconstruction accuracy. We also test the influence of PbGMR-GMUS and the attention conditional flow model to the final performance of the deterministic task, as shown in Table 9. The result shows that both the encoder-decoder part and the flow model are necessary for the success of the whole framework.
Table 8: The reconstruction error of stochastic flow with the same training epoch number for Backward-facing step flow dataset. The residual connection and centers can decrease the reconstruction error significantly.
residual connection	centers	v
x	x	85
x	✓	19.2
✓	✓	15.8
Table 9: Ablation study: the average relative rollout error with/without each proposed component for vascular flow. CNF: using conditional normalizing flows for sequence prediction. In variant 2, we remove the CNF block and use a transformer only structure for predictions, similar to [11]. Our model with residual connection and centers have the best performance.
residual connection+centers	CNF	u	v	p
GMR-GMUS	X	X	7.3	10	22
Variaint1	X	✓	6.1	8.7	9.2
Variaint2	✓	X	5	8	3
Ours	✓	✓	3.49	5.47	1.05
To efficiently incorporate the temporal information, we conceive a temporal conditional normalizing flows to account for the long-time temporal dependency. The original RealNVP is only a probabilistic model, and it is not straightforward to include temporal information. To achieve this, we propose an encoding-decoding transformer structure to incorporate fixed physical parameters and all previous steps into condition vector  $c$ . In this ablation study, we remove the transformer structures and replace condition vector  $c$  by the current step latent vector  $z_{t-1}$  when predicts next step  $z_t$ . This variant can be treated as a probabilistic 1-step prediction method. The result in Table 10 shows that the proposed structure can significantly improve the accuracy compared with such 1-step method, while still easy to calculate the probability of a coupling layer if we concatenate condition vector at every coupling layer, as shown in Appendix A.1.
Table 10: Ablation study: effect of attention-based temporal conditional model for Sonic dataset, NF: normalizing flows only condition on previous step. The proposed model outperforms 1 step NF, indicating the design of attention based temporal model is necessary for accurate time series prediction.
u	v	p	T
1-step NF	4.85	16.36	1.09	5.50
Ours	0.37	0.85	0.079	0.01
Our model can benefit from longer time dependencies, since we take all previous steps in the experiment setting. Also, we can use the moving window to reduce the cost further. We did one additional experiment of reducing the number of inputs during the inference time, as shown in Table 11. When reducing the window size from 400 to 150, the degradation in accuracy is not significant. So the model is also applicable to cases with thousands or even longer steps with the sliding window.
Table 11: Ablation study: effect of window length (WL) for Cylinder dataset, reducing the window length to 150 doesn't increase error significantly.
WL	u	v	p
150	4.14	80.48	22.99
400	3.8	74	20
Though we only tested RealNVP in the experiment part, other normalizing flows are also feasible here. In Table 12, we test another normalizing flow model MAF [54], and find the result is comparable to RealNVP. It indicates that the proposed framework is flexible.
Table 12: Ablation Study: the average rollout error using different normalizing flow models. MAF: Masked Autoregressive Flow. The performance is comparable with RealNVP.
Dataset-rollout step
Variable	Cylinder flow-400	Sonic flow-40	Vascular flow-250
u	v	p	u	v	p	T	u	v	p
Ours	RealNVP	3.8	74	20	0.37	0.85	0.079	0.01	3.49	5.47
Ours	MAF	3.8	72	19.13	0.33	0.69	0.055	0.02	3.82	5.84
As a probabilistic model, the proposed PbGMR-GMUS + conditional flow model is trying to fit such a stochastic process and get various samples with the same physical statistics. To demonstrate these, we test MGN and GMR-GMUS on the stochastic dataset. In Table 13, we can see that the performance of the proposed model is better than other learning-based models on all metrics. In Figure 20, we visualize the mean and variance of velocity from each model, and find the MGN can not capture the real distribution at all. The result of GMR-GMUS model, though looks reasonable in statistics, can only produce the same output given the same input condition, which doesn't reflect the stochasticity of the last dataset. In Figure 21, we get two samples from GMR-GMUS, and find the two samples are exactly the same.
Table 13: Criteria of generation quality for the stochastic flow for URANS and different learning-based models, the proposed model has the best performance.
CRPS (C_u,↓)	CRPS (C_v,↓)	FVD (d_u,↓)	FVD (d_v,↓)	MFE (e_u,↓)	MFE (e_v,↓)	TE (TKE,↑)
URANS	3.24	2.0	228112	137860	0.31	0.94	0.192
MeshGraphNet	5.06	2.47	1159950	179206	0.82	1.60	0.87
GMR-GMUS	2.57	2.27	4976	3740	0.167	1.96	0.99
Ours	1.28	1.08	1262	397	0.0176	0.176	0.99
Figure 20: Evaluation lines of mean (left) and variance (right) of streamwise velocity  $u$ :  $x$ -evaluation lines  $(- - -)$ . Quantities from LES  $(\text{—})$ , URANS  $(\text{—})$ , MeshGraphNet  $(\text{—})$ , GMR-GMUS  $(\text{—})$  and the proposed model  $(\text{—})$
Figure 21: Velocity magnitude at 3 time steps 0, 50, 100 (left to right) from different samples using GMR-GMUS + transformer for the stochastic dataset. 1st row for sample 1. 2nd row for sample 2. The deterministic model can only produce the same output while the proposed model can generate different realizations.