## 4 Results
### Performance
#### 4.2.2 LBM Performance
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Maximum**} & \multicolumn{4}{c}{**Average**} \\ \cline{2-9}
**Layout** & **Mesh** & \(P_{\text{LUPS}}\) & \(P_{\text{B}}\) & \(U_{\text{B}}\) & \(P_{\text{LUPS}}\) & \(P_{\text{B}}\) & \(U_{\text{B}}\) \\ \hline f32 dense & \(1024^{2}\) & 5.37 & 386 & 0.705 & 5.12 & 369 & 0.674 \\ f32 bitmask node & \(2048^{2}\) & 5.13 & 370 & 0.675 & 4.78 & 344 & 0.629 \\ f32 tile & \(1024^{2}\) & 5.12 & 369 & 0.673 & 4.53 & 326 & 0.596 \\ f32 pointer tile & \(1024^{2}\) & 4.12 & 296 & 0.541 & 3.85 & 277 & 0.506 \\ f64 dense & \(2048^{2}\) & 2.65 & 381 & 0.696 & 2.51 & 362 & 0.661 \\ f64 bitmask node & \(3955^{2}\) & 2.60 & 375 & 0.685 & 2.40 & 346 & 0.631 \\ f64 tile & \(1024^{2}\) & 2.58 & 372 & 0.679 & 2.37 & 341 & 0.622 \\ f64 pointer tile & \(2509^{2}\) & 2.46 & 354 & 0.646 & 2.42 & 348 & 0.635 \\ \hline \multicolumn{9}{l}{**Other work**} \\ \hline
[4] D3Q19, f64, K40 & \(243^{3}\) & 0.77 & 235 & 0.816 & - & - & - \\
[4] D3Q19, f32, K40 & \(243^{3}\) & 1.48 & 226 & 0.782 & - & - & - \\
[3] D3Q19, f32, K20c & \(192^{3}\) & 1.04 & 158 & 0.757 & 0.98 & 149 & 0.714 \\
[16] D3Q19, f64, TITAN & - & 0.64 & 194 & 0.674 & - & - & - \\
[17] D3Q19, f32, Titan Xp & \(100^{3}\) & 2.20 & 334 & 0.609 & - & - & - \\
[17] D3Q19, f64, Titan Xp & \(252^{3}\) & 0.99 & 301 & 0.550 & - & - & - \\
[16] D2Q9, f64, TITAN & - & 1.02 & 147 & 0.509 & - & - & - \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Performance of cavity simulations for different data layouts. Performance \(P_{\text{LUPS}}\) is given in GLUPS, \(P_{\text{B}}\) in GB/s. Column \({}^{\prime}\)Mesh\({}^{\prime}\) contains mesh size for which maximum performance was observed. Average performance is computed for meshes containing at least \(1024^{2}\) nodes. Reference performances for other work are adapted from [3; 4; 16; 17] and contain results for collision operators similar to the one used in this paper. For GPUs different to GTX Titan Xp, only \(U_{\text{B}}\) values can be directly compared. The works of [3; 4] use optimised dense layouts, whereas [16; 17] employ tiles. Figure 6: Performance of cavity GPU simulation for different mesh sizes, memory layouts and data types (f32 and f64 denote single- and double-precision floating-point numbers, respectively). We compared performance for four different data layouts. The dense layout is a standard, multidimensional array containing all PDFs for all nodes. As excepted, this version offers the highest performance to 70% of peak GPU memory bandwidth. This result is close to the best-reported values for highly optimised codes from [3,4] and is consistent with performance reported for the same hardware in [17]. During code profiling, we observed that loads of all PDFs (excluding \(f_{0},f_{2}\) and \(f_{5}\)) cause uncoalesced transactions because, as shown in [4], neighbour \(f_{i}\) values are shifted in memory by one position. We found no method to correct this behaviour--the typical technique is the usage of shared memory but the Taichi language offers no such feature. The bitmasked data structures can be used in many ways. At first, we applied a single bitmask per each \(f_{i}\) function because, due to the structure-of-arrays data layout, we were not able to apply a single bitmask per a whole lattice node. The observed performance dropped more than twice compared to the dense layout--maximum bandwidth was at the level of 140 GB/s for the single-precision version. However, the bitmask layout does not save memory and serves only as a convenient way to skip computations for non-existent data. The reasonable way is then to use bitmasks only for a field containing encoded node type. This layout enables for simple management of sparse geometries and is marked as \"bitmask node\" on performance plots. As can be seen in Figure 6 and Table 2, when a single bitmask is used per whole data of a single lattice node, the performance loss is less than 10% compared to the dense layout for geometries containing at least \(10^{6}\) nodes.