# LLVM Compiler/Installation
## Installing Pre-build Binaries

on Ubuntu
* sudo apt-get install clang llvm

## Installing from github master

Hardware/Operating system
* Azure VM | Standard NC6_Promo (6 vcpus, 56 GiB memory)
* Tesla K80, which has compute capability 3.7. You can find out what card you got via lshw -C display; you can find out the compute capability of your card https://developer.nvidia.com/cuda-gpus .
* Linux (ubuntu 18.04)

### Prerequisites

Install CUDA 10.2 , using instructions https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork
After installation, export two environment variables, vim ~/.bashrc to add two lines
* export PATH=$PATH:/usr/local/cuda-10.2/bin/
* export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/lib/

### Get source packages

Assuming you are in your home directory
cd
git clone https://github.com/llvm/llvm-project.git

### Build using GCC

Explanation for configuration options
* -DCMAKE_C_COMPILER=gcc // the C compiler used to compile clang/llvm, GCC
* -DCMAKE_CXX_COMPILER=g++ // the C++ compiler used to compile clang/llvm: G++
* -DLLVM_TARGETS_TO_BUILD=X86;PowerPC;NVPTX;AMDGPU // explicitly specify target devices to support, Intel ,Nvidia, IBM, and AMD CPUs or GPUs
* -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_70 // default GPU computing capability version to support, https://developer.nvidia.com/cuda-gpus lists such information.
* -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=37,60,70 // all GPU computing capability versions to build in libomptarget
Other optional options
* -DGCC_INSTALL_PREFIX=${GCC_PATH}
* -DCMAKE_C_COMPILER=${GCC_PATH}/bin/gcc
* -DCMAKE_CXX_COMPILER=${GCC_PATH}/bin/g++
* -DCMAKE_Fortran_COMPILER=${GCC_PATH}/bin/gfortran
* -DCUDA_PATH= // this option should be automatically set if cuda is in your search path
* -DCUDA_TOOLKIT_ROOT_DIR= // this option should be automatically set if cuda is in your search path
* -DOPENMP_ENABLE_LIBOMPTARGET=ON // this should be on by default
* -DLIBOMP_FORTRAN_MODULES=ON
* -DBUILD_SHARED_LIBS=OFF // turn off shared libs
More examples
* cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;lld;openmp" -DCMAKE_INSTALL_PREFIX=/Users/abc/llvm-research/inst-10.0.1 -DCMAKE_BUILD_TYPE=Debug ../llvm/
* ninja -j8
* ninja install

### Rebuild using Clang

add the path to the installed clang, vim ~/.bashrc
Optional options, explictly turn on bitcode lib, and the compiler/linker to build it
* -DLIBOMPTARGET_NVPTX_ENABLE_BCLIB=true
* -DLIBOMPTARGET_NVPTX_CUDA_COMPILER=${PREFIX}/bin/clang
* -DLIBOMPTARGET_NVPTX_BC_LINKER=${PREFIX}/bin/llvm-link

## Installing from releases

Hardware/Operating system
* Azure VM | Standard NC6_Promo (6 vcpus, 56 GiB memory)
* Tesla K80, which has compute capability 3.7. You can find out what card you got via lshw -C display; you can find out the compute capability of your card here.
* Linux (ubuntu 18.04)

### Prerequisites

Install CUDA 10.2 , using instructions https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork
Assuming you current path is
* /home/ubuntu/omp5-gpu-llvm
After installation, export two environment variables, vim ~/.bashrc to add two lines
* export PATH=$PATH:/usr/local/cuda-10.2/bin/
* export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/lib/

### Get source packages

Three steps to download, untar, and put them into the right locations.
In the end, the directory layout should look like
* llvm-10.0.0.src
** tools/clang
** projects/openmp
** projects/compiler-rt

### Build the Compiler with OpenMP offloading support

You need to know the Compute Capability version of your GPU. https://developer.nvidia.com/cuda-gpus lists such information. For example, some typical GPUs and their CC versions are:
* Tesla K80 3.7, sm_37
* Tesla P100 6.0, sm_37
* Tesla V100 7.0, sm_37
After the installation, you should expand your PATH and LD_LIBRARY_PATH again
* export PATH=$PATH:/home/ubuntu/omp5-gpu-llvm/install/bin
* export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/ubuntu/omp5-gpu-llvm/install/lib

### Rebuild the OpenMP runtime libraries

You should use the freshly installed clang to rebuild the OpenMP runtime library

## Test the installation

save the following code into a file named ongpu.c
Compile and run it

## Troubleshooting
### error while loading libomp.so

./a.out: error while loading shared libraries: libomp.so: cannot open shared object file: No such file or directory solution
* export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/pathTo/installed-llvm/lib

## External Links

* OpenMP 5 GPU offloading instructions [Fabio]: https://github.com/devitocodes/devito/wiki/OpenMP-5-GPU-offloading-instructions--%5BFabio%5D
** Clang_with_OpenMP_Offloading_to_NVIDIA_GPUs: https://hpc-wiki.info/hpc/Building_LLVM/Clang_with_OpenMP_Offloading_to_NVIDIA_GPUs
* OpenMP 5 GPU offloading instructions with Clang 11 [Italo] https://github.com/devitocodes/devito/wiki/OpenMP-5-GPU-offloading-instructions-with-Clang-11-%5BItalo%5D
* a docker file showing LLVM installation: https://github.com/freeCompilerCamp/play-with-compiler/blob/master/dockerfiles/dind/Dockerfile.llvm10