Fully-fused Multi-Layer Perceptrons on Intel Data Center GPUs
Authors: name: Kai Yuan, name: Christoph Bauinger, name: Xiangyi Zhang, name: Pascal Baehr, name: Matthias Kirchhart, name: Darius Dabert, name: Adrien Tousnakhoff, name: Pierre Boudier, name: Michael Paulitsch

This paper presents a SYCL implementation of Multi-Layer Perceptrons (MLPs),
which targets and is optimized for the Intel Data Center GPU Max 1550. To
increase the performance, our implementation minimizes the slow global memory
accesses by maximizing the data reuse within the general register file and the
shared local memory by fusing the operations in each layer of the MLP. We show
with a simple roofline model that this results in a significant increase in the
arithmetic intensity, leading to improved performance, especially for
inference. We compare our approach to a similar CUDA implementation for MLPs
and show that our implementation on the Intel Data Center GPU outperforms the
CUDA implementation on Nvidia's H100 GPU by a factor up to 2.84 in inference
and 1.75 in training. The paper also showcases the efficiency of our SYCL
implementation in three significant areas: Image Compression, Neural Radiance
Fields, and Physics-Informed Machine Learning. In all cases, our implementation
outperforms the off-the-shelf Intel Extension for PyTorch (IPEX) implementation
on the same Intel GPU by up to a factor of 30 and the CUDA PyTorch version on
Nvidia's H100 GPU by up to a factor 19. The code can be found at
https://github.com/intel/tiny-dpcpp-nn.