An Introduction to GPU Optimization | by Anuradha Wickramarachchi | Towards Data Science

Most of the tasks that involve heavy computations take time and it becomes further time consuming as the datasets get bigger and bigger. One way of addressing this problem is using threads. In this article, I will be introducing GPU acceleration, how it is done and a simple API for GPU tasks along with a code. To start with, let us consider the example of a matrix multiplication.
In the above scenario, there are 2 matrices one of size 3 x 6 and another of size 6 x 6. The resulting matrix will be of size 3 x 6. Therefore, for each cell of the matrix there will be 6 calculations. In total, there will be 3 x 6 x 6 many multiplications. Hence, we can conclude that the task takes O(mn2) amount of time for the calculation. This implies that, for a matrix of size 2000 x 2000 there will be 8,000,000,000 calculations done. This is a LOOOOT of CPU time!!!.
Usually GPUs contain a large number of processing cores. Generally varying from 384 up to several thousands. Below is a simple comparison of some of the latest graphics cards of nvidia. (Source)
CUDA stands for Compute Unified Device Architecture. These are running at relatively low speeds but provides greater parallelism by employing a large number of ALUs (Arithmetic and Logical Units). Read more here.
This diagram demonstrates the threading model in CUDA (this is quite the same as other architectures in the market such as AMD). To make it simple to start with, we may assume that each CUDA core or a GPU core can run a thread at a time. If we have a large data set, we can break it into pieces. A Grid contains several Blocks, and Block is another matrix which contains a number of threads equal to the size of it. Anyhow, as this is the introduction let’s focus on the bigger picture with a simpler API developed using JAVA.
As we have discussed, each GPU core is capable of running a separate thread. The simplest way to start the analogy is assuming that each cell of the matrix will be calculated by a single GPU core. And since all the cores are running in parallel, all the cells will be computed in parallel. Therefore, our time complexity suddenly drops to O(n). Now, for the 2000 x 2000 matrix we just need 2,000 runs, which is quite easy for a computer to compute. Usually each the threads we discussed before, knows its identity, which is the Block and the Grid it belongs to. Or in more simpler words the cell location of the matrix. Also, the matrix will be loaded into the shared memory of the GPU where we can directly access cell data by indices and process in parallel. Easy right? Let’s check on the code.
What? Well, APARAPI (A-PARallel-API) is a JAVA wrapper for OpenCL which is the Open Computing Language used for programming GPUs. This supports both CUDA architecture and AMD devices. Also, the API bring in the greater object orientation of JAVA into the picture, which might look like a mess if we directly jump in to the task with C++. Getting started is quite easy. There is a maven dependency. But make sure you have correctly setup OpenCL or CUDA. Simple googling should help you with how. Most of the devices come with them bundled in (OSX and Windows devices).
Kernel is the part of the code that is being executed with the GPU. The variables that are visible to the Kernel will be copied to the GPU RAM. We are feeding data in terms of linear arrays rather than 2D arrays as this is the way supported by the GPUs. Not that they are not capable of handling 2D arrays, but the way they are handled is through the concept of dimensions (We won’t talk it just yet).
Range range = Range.create(SIZE * SIZE);
The above code allocates memory in the GPU so that SIZE x SIZE amount of threads or less (As available) running in the GPU.
int row = getGlobalId() / SIZE;int col = getGlobalId() % SIZE;
The above code obtains the Id of the thread from its private memory. With this for that particular thread, we can identify the thread’s cell location. For each cell we would do the following.
for (int i = 0; i < SIZE; i++) {    d[row * SIZE + col] += a[row * SIZE + i] * b[i * SIZE + col];}
This is the simple sum of multiples of the corresponding cells of the two matrices. We just define the Kernel for a single thread using thread indices, which will be run in parallel for all the threads.
It is fast. But how fast?. This is the output of the above program.
1200 x 1200
Starting single threaded computationTask finished in 25269ms Starting GPU computationTask finished in 1535ms
Following was run only for the GPU component since times were quite large for CPU to compute.
2000 x 2000 Task finished in 3757ms5000 x 5000 Task finished in 5402ms
Hope you enjoyed reading. Please do try!! Cheers!!