## 4 Results
### Performance  
#### 4.2.1 Memory Bandwidth  
Before analysis of the implemented kernel, we first measured available memory bandwidth during a simple copy of data between one-dimensional arrays for different data layouts implemented in Taichi. The results are shown in Figure 5. Notice that we use both SI (\(\mathrm{k}=10^{3}\), \(\mathrm{M}=10^{6}\), \(\mathrm{G}=10^{9}\)) and binary (\(\mathrm{Ki}=2^{10}\), \(\mathrm{Mi}=2^{20}\), \(\mathrm{Gi}=2^{30}\)) prefixes. As an approximate reference, we run the NVidia _bandwidthTest_ utility. Internally, _bandwidthTest_ measures calls to the _cudaMemcpy_ function, but it should be noted that this utility does not use preliminary \"warm up\" of the measured code. Furthermore, the _bandwidthTest_ has no support for size arguments larger than \(2^{31}\) bytes, and the results are displayed in MiB/s and must be scaled. For Taichi, we prepared a simple kernel copying data from one one-dimensional array to the other. The number of threads per thread block was explicitly set to 512 since it resulted in high average performance. Time duration measurements were conducted using Python _time.perf_counter_ for 100 kernel calls (as in _bandwidthTest_). Additionally, before each measurement, we initially called the measured kernel five times to force runtime compilation and initialise contents of cache memories and translation lookaside buffers (TLB). To investigate the strange behaviour observed in Figure 5 for memory block sizes around 1 MiB, we prepared an additional kernel shown in Listing 1. We used custom kernel copying data because it is difficult to observe internal implementation details of the _cudaMemcpy_ function. For a large amount of transferred data, the performance of the simple kernel is slightly lower than the performance of _cudaMemcpy_, but for small memory blocks, the copy kernel takes less time. We can thus suppose that _cudaMemcpy_ uses an optimised method of data copying but introduces more overhead than a simple kernel launch. When small data blocks are copied, for both data copying methods, the performance increases with block size due to a growing ratio of transfer time to overheads (transfer time grows with memory block size). Because of a high cache hit ratio, it is possible to obtain bandwidth even higher than the maximum memory bandwidth--for 1,383,048 bytes block size, the simple kernel achieved up to 663 GB/s and L2 cache hit ratio about 0.5. If the  
Figure 4: Velocity magnitude \(|\mathbf{v}|\) and streamlines (white) after \(10^{6}\) time steps for flows past a cylinder at different viscosities \(\
u\). The picture on the top shows a separation bubble comprising two symmetric and counter-rotating recirculation zones. Other images contain unsteady Kármán vortex street patterns. At the inlet, constant velocity is set to \(v_{x}=0.1\). Mesh resolution is 4096 \(\times\) 512 nodes, sphere diameter and square edge are equal to half of the channel height. The pointer layout has the lowest maximum measured bandwidth, although it is less dependent on the size of the block of data assigned to a single pointer than the bitmask layout. For a single pointer per 8 bytes of data, the bandwidth was below 20 GB/s and, additionally, we had to significantly decrease the amount of allocated memory. With the increasing size of the pointed data block, the bandwidth steadily increased, achieving 247 GB/s at 124 MiB transferred data block for a single pointer per 512 bytes of data, as shown in Figure 5. Then, when increasing the amount of data per single pointer, the maximum bandwidths stayed in the 230-250 GB/s range for up to 16 MiB of data per pointer. After this limit, the bandwidth dropped to 144 GB/s when a single pointer was used for a block containing 1 GiB of data. We can also observe that the pointer layout has different overheads than other data layouts--for the small size of the data block, the performance plot for the pointer layout has a different shape than for other layouts.