## 2 Theory
### The Roofline Model  
Initially proposed by [PERSON] et al. in 2008, the roofline model [4] is a throughput-oriented performance model. It aims to visualize the performance \(P\) of a code on a specific micro-architecture. The model provides insights on how to improve the overall performance of a given program, on its main bottlenecks, and can provide the developer a vision of the most adapted architecture to his code.  
The roofline model relies on three main concepts:  
* The peak performance \(P_{m}\) of the CPU/GPU, which can be defined as the maximum speed at which all the components of a machine can operate [26]. In a considerable number of application codes, this limit is never reached. In this work, \(P_{m}\) is expressed in terms of GFLOP/s.
* The main memory bandwidth \(B_{s}\), which is the rate at which data can be transferred from memory to CPU/GPU [26]. It is expressed in GB/s. In several scientific programs and modeling applications, DRAM bandwidth is considered the main bottleneck. Memory specifications refer to \(B_{s}\) that a certain memory chip can achieve. However, this theoretical bandwidth requires perfect circumstances and is not attainable in reality [2].
* The operational intensity \(I\) of a program; this metric characterizes the code itself, while the machine is characterized by \(P_{m}\) and \(B_{s}\). \(I\) is defined as the ratio between the number of operations performed by a given kernel and the number of bytes of memory transferred from a memory level to the computing unit. In our case, the operations refer to Floating Point additions and multiplications. Hence, the operational intensity is expressed in FLOP/byte.  
The graph of the roofline model is on a log-log scale, the X-axis is FLOP/byte, and the Y-axis is GFLOP/s. The peak Floating Point performance \(P_{m}\) of the machine is expressed as a horizontal line or ceiling. The peak memory performance is considered as the maximum FLOP performance that the memory can provide for a specific value of the operational intensity \(I\). The peak memory performance is represented by a line at a 45-degree angle. The two ceilings intersect at the point of peak FLOP performance and peak DRAM bandwidth [4]. They also provide what we call the performance limits by setting upper bounds to the attainable performance of a kernel; see Figure 3. Hence, the performance \(P\) verifies the following inequality.  
\[P\leq\min{(P_{m},I\times B_{\rm s})}. \tag{3}\]  
The architecture efficiency \(e_{i}\) of an application \(a\) running on an architecture \(i\) is provided by  
\[e_{i}(a)=\frac{P}{min(P_{m},I\times B_{\rm s})}. \tag{4}\]  
Conventionally, the roofline model focuses on one level in the memory hierarchy (DRAM or HBM). However, this has been extended to the full memory system, DRAM, HBM, and caches, in order to understand the impact of data locality and reuse. This model provides additional insights into the performance of the memory system [27; 28]; see Figure 4.  
Figure 3: Sketch of the roofline model. The red lines present performance boundaries. Using the model, kernels are subdivided into two categories: kernels on the left of the green dashed line, which are memory bound, and others on the right of this line are said to be compute bound. Program 1 is memory bound. It is mainly limited by data traffic and memory access. Programs 2 and 3 are compute bound. Program 3 is close to the theoretical upper bound, which means that it uses the micro-architecture in an optimal manner. However, programs 1 and 2 are far from their boundary, not adequately using the micro-architecture.