# LLVM Compiler/Printable version

The current, editable version of this book is available in Wikibooks, the open-content textbooks collection, at
https://en.wikibooks.org/wiki/LLVM_Compiler

# Installation
## Installing Pre-build Binaries

on Ubuntu
* sudo apt-get install clang llvm

## Installing from github master

Hardware/Operating system
* Azure VM | Standard NC6_Promo (6 vcpus, 56 GiB memory)
* Tesla K80, which has compute capability 3.7. You can find out what card you got via lshw -C display; you can find out the compute capability of your card https://developer.nvidia.com/cuda-gpus .
* Linux (ubuntu 18.04)

### Prerequisites

Install CUDA 10.2 , using instructions https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork
After installation, export two environment variables, vim ~/.bashrc to add two lines
* export PATH=$PATH:/usr/local/cuda-10.2/bin/
* export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/lib/

### Get source packages

Assuming you are in your home directory
cd
git clone https://github.com/llvm/llvm-project.git

### Build using GCC

Explanation for configuration options
* -DCMAKE_C_COMPILER=gcc // the C compiler used to compile clang/llvm, GCC
* -DCMAKE_CXX_COMPILER=g++ // the C++ compiler used to compile clang/llvm: G++
* -DLLVM_TARGETS_TO_BUILD=X86;PowerPC;NVPTX;AMDGPU // explicitly specify target devices to support, Intel ,Nvidia, IBM, and AMD CPUs or GPUs
* -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_70 // default GPU computing capability version to support, https://developer.nvidia.com/cuda-gpus lists such information.
* -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=37,60,70 // all GPU computing capability versions to build in libomptarget
Other optional options
* -DGCC_INSTALL_PREFIX=${GCC_PATH}
* -DCMAKE_C_COMPILER=${GCC_PATH}/bin/gcc
* -DCMAKE_CXX_COMPILER=${GCC_PATH}/bin/g++
* -DCMAKE_Fortran_COMPILER=${GCC_PATH}/bin/gfortran
* -DCUDA_PATH= // this option should be automatically set if cuda is in your search path
* -DCUDA_TOOLKIT_ROOT_DIR= // this option should be automatically set if cuda is in your search path
* -DOPENMP_ENABLE_LIBOMPTARGET=ON // this should be on by default
* -DLIBOMP_FORTRAN_MODULES=ON
* -DBUILD_SHARED_LIBS=OFF // turn off shared libs
More examples
* cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;lld;openmp" -DCMAKE_INSTALL_PREFIX=/Users/abc/llvm-research/inst-10.0.1 -DCMAKE_BUILD_TYPE=Debug ../llvm/
* ninja -j8
* ninja install

### Rebuild using Clang

add the path to the installed clang, vim ~/.bashrc
Optional options, explictly turn on bitcode lib, and the compiler/linker to build it
* -DLIBOMPTARGET_NVPTX_ENABLE_BCLIB=true
* -DLIBOMPTARGET_NVPTX_CUDA_COMPILER=${PREFIX}/bin/clang
* -DLIBOMPTARGET_NVPTX_BC_LINKER=${PREFIX}/bin/llvm-link

## Installing from releases

Hardware/Operating system
* Azure VM | Standard NC6_Promo (6 vcpus, 56 GiB memory)
* Tesla K80, which has compute capability 3.7. You can find out what card you got via lshw -C display; you can find out the compute capability of your card here.
* Linux (ubuntu 18.04)

### Prerequisites

Install CUDA 10.2 , using instructions https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork
Assuming you current path is
* /home/ubuntu/omp5-gpu-llvm
After installation, export two environment variables, vim ~/.bashrc to add two lines
* export PATH=$PATH:/usr/local/cuda-10.2/bin/
* export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/lib/

### Get source packages

Three steps to download, untar, and put them into the right locations.
In the end, the directory layout should look like
* llvm-10.0.0.src
** tools/clang
** projects/openmp
** projects/compiler-rt

### Build the Compiler with OpenMP offloading support

You need to know the Compute Capability version of your GPU. https://developer.nvidia.com/cuda-gpus lists such information. For example, some typical GPUs and their CC versions are:
* Tesla K80 3.7, sm_37
* Tesla P100 6.0, sm_37
* Tesla V100 7.0, sm_37
After the installation, you should expand your PATH and LD_LIBRARY_PATH again
* export PATH=$PATH:/home/ubuntu/omp5-gpu-llvm/install/bin
* export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/ubuntu/omp5-gpu-llvm/install/lib

### Rebuild the OpenMP runtime libraries

You should use the freshly installed clang to rebuild the OpenMP runtime library

## Test the installation

save the following code into a file named ongpu.c
Compile and run it

## Troubleshooting
### error while loading libomp.so

./a.out: error while loading shared libraries: libomp.so: cannot open shared object file: No such file or directory solution
* export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/pathTo/installed-llvm/lib

## External Links

* OpenMP 5 GPU offloading instructions [Fabio]: https://github.com/devitocodes/devito/wiki/OpenMP-5-GPU-offloading-instructions--%5BFabio%5D
** Clang_with_OpenMP_Offloading_to_NVIDIA_GPUs: https://hpc-wiki.info/hpc/Building_LLVM/Clang_with_OpenMP_Offloading_to_NVIDIA_GPUs
* OpenMP 5 GPU offloading instructions with Clang 11 [Italo] https://github.com/devitocodes/devito/wiki/OpenMP-5-GPU-offloading-instructions-with-Clang-11-%5BItalo%5D
* a docker file showing LLVM installation: https://github.com/freeCompilerCamp/play-with-compiler/blob/master/dockerfiles/dind/Dockerfile.llvm10

# Clang Plugins
## How to Debug a Plugin

There are two ways to run a Clang plugin:
* call the clang frontend directly: clang -cc1 ...
* call the compiler driver, clang -v, to find out real command lines

# OpenMP Support

With the release of Clang 3.8.0, OpenMP 3.1 support is enabled in Clang by default, and the OpenMP runtime is therefore built as a normal part of the Clang build, and distributed with the binary distributions. You do not, therefore, need explicitly to check out this code, or build it out of tree; a normal Clang check out and build will automatically include building these runtime libraries.
https://openmp.llvm.org/

## installation

cmake -G Ninja -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=$LLVM_PATH -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;compiler-rt;openmp" -DCLANG_BUILD_EXAMPLES=1 $LLVM_SRC/llvm
ninja install -j8 -l8

## OpenMP lowering or Code Geneneration
### Hello example

clang -fopenmp -S -O3 -emit-llvm omp_hello.c
cat omp_hello.ll

### offloading example

clang -fopenmp -S -O3 -emit-llvm -fopenmp-targets=nvptx64-nvidia-cuda offloading_success.cpp

## relevant source files

List
* tests: clang/test/OpenMP
* Clang CodeGen for OpenMP IR:
** clang/lib/CodeGen/CGOpenMPRuntime.cpp
** clang/lib/CodeGen/CGStmtOpenMP.cpp //This contains code to emit OpenMP nodes as LLVM code.
* LLVM OMP IR Builder: llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
* Runtime support: openmp/runtime/src/kmp_csupport.cpp

## How to Debug

First, use -v to show the actual command line used by the clang driver:
clang -v -fopenmp omp_hello.c
Next, use gdb to debug the actual executable with the full command line options
* gdb -args /home/ubuntu/install/llvm_install/bin/clang-10 -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -main-file-name omp_hello.c -mrelocation-model static -mthread-model posix -mframe-pointer=all -fmath-errno -fno-rounding-math -masm-verbose -mconstructor-aliases -munwind-tables -target-cpu x86-64 -dwarf-column-info -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/ubuntu/install/llvm_install/lib/clang/10.0.1 -internal-isystem /usr/local/include -internal-isystem /home/ubuntu/install/llvm_install/lib/clang/10.0.1/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir /home/ubuntu -ferror-limit 19 -fmessage-length 0 -fopenmp -fgnuc-version=4.2.1 -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -faddrsig -o /tmp/omp_hello-c73985.o -x c omp_hello.c
Warmup run of the compiler within gdb
* (gdb) r
Set a breakpoint at 1829 llvm::FunctionCallee CGOpenMPRuntime::createRuntimeFunction(unsigned Function)
* b CGOpenMPRuntime.cpp:1829
* r # run it
Now can check the backtrace/call stack