## 4 Results
### Performance
#### 4.2.2 LBM Performance
For smaller geometries, the bitmask layout has low performance. An additional advantage of this approach is that, due to the fine-grained masking of single nodes, only valid nodes are processed, even for very complex geometries. The pointer layout available in Taichi can also be used in many ways, but the applied method should allow storage in memory only values used during computations. We then used a single pointer per _tile_ containing data for \(16^{2}\) neighbour nodes. The resulting data layout is denoted as \"pointer tile\". Additionally, we measured performance for the \"tile\" layout defined as a dense array of tiles without an additional layer of pointers. The measured performance of the tile layout was similar to the dense layout, but only for geometries with less than \(2048^{2}\) nodes. After this limit, the achieved bandwidth utilisation dropped even below 0.5 for single-precision data and the largest geometry. We observed two issues appearing for the tile data layout. First, the dimension of the CUDA thread block had to be reduced to the number of nodes per tile--in our case from 512 to 256 threads per block. For the dense layout, such a change of thread block size decreased bandwidth from 360 to 338 GB/s for the cavity \(4096^{2}\) case and single-precision data. Next, the Taichi has no support for low-level optimisations presented in [16,17], e.g., usage of shared memory, warp level programming, and LBM-optimised index calculations inside a tile. For example, we tried to store \(f_{1}\) and \(f_{3}\) functions using column-major order, but we encountered different behaviour than described in Taichi documentation. It is also probable that the observed decrease in performance may be caused by other, as yet undetected reasons. For the pointer tile layout, the performance is slightly surprising. When double-precision data are used, then the performance stays high and steady even for the largest geometries, despite the drop-out observed for tile layout. On the other hand, performance for single-precision is also almost constant but at the low level (\(U_{B}=0.5\)) given by the minimum value observed for tile layout and the largest geometry. We have not found the cause of such behaviour yet and only found that the code handling pointers in the kernel is quite complex and significantly increases register pressure--the kernel required 70 registers which decreased the theoretical occupancy to 37.5%. Performance for sparse geometries is measured in the same way as for dense ones, but only non-solid nodes are taken into account in the performance calculation. Thus, for each sparse geometry, we define its _porosity_  
\[\phi=\frac{n_{\text{non-solid nodes}}}{n_{\text{all nodes}}} \tag{8}\]  
which determines what proportion of all nodes are involved in the LBM calculations. We treat all non-solid nodes as computational because LBM implementation is bandwidth-bound on our system and, even for the bounce-back boundary nodes, which do not require computations, we need to read and write \(f_{i}\) functions from/to memory. The performance for sparse geometries was measured for square geometries with \(4096^{2}\) nodes and different porosities \(\phi\geq 0.1\). The large geometry size was chosen to keep at least \(10^{6}\) non-solid nodes, even for the lowest porosity. To obtain the required \(\phi\), the geometry was filled with solid, circle obstacles. We used two different arrangements of obstacles: a regular array and a random placement, with examples shown in Figure 7. The regular array contains a mesh of \(8\times 8\) circles, of which the radius depends on the required \(\phi\). We did not use regular arrays for \(\phi<0.3\) because, in such cases, all geometry walls are filled with solid nodes from overlapping circles. For the random placement, the geometry was filled with randomly placed circles. The radii of circles were also randomly chosen from \(r\in[8,256]\) nodes.