Xen ARM with Virtualization Extensions/Salvator-XS

= General information =
This Wiki describes how to run system with Xen and Dom0 on Renesas Salvator-XS board with R-Car H3 ES3.0 SoC (Arm64).
Main information how to deal with Salvator-XS board located at:
 https://elinux.org/R-Car/Boards/Salvator-XS
Please note, although Wiki only covers Salvator-XS + H3 ES3.0 configuration, the R-Car M3 ES3.0 (M3-W+) SoC is also supported. It is possible to run Xen on other "board & SoC" configurations with updating platform specific parts (device-tree, etc), the hypervisor part will remain the same. For example, the following configurations will work as well:
 https://elinux.org/R-Car/Boards/M3SK
 https://elinux.org/R-Car/Boards/H3SK
Due to the fact that Renesas provides their reference software in form of Yocto BSP build instruction, this Wiki provides additional steps in order to build and run system with Xen and Dom0, it relies on Renesas Yocto BSP v5.1.0
= Build BSP =
* Follow the build instruction in Manual steps mode up to the build step #4, including it. 
* Clone meta-renesas layer from xen-troops and cherry-pick last 4 patches from branch v5.1.0_xen, this is the minimum set of patches needed to run system with Xen and Dom0. Please note, all local changes need to be stashed or committed:
 cd ${WORK}/meta-renesas
 git remote add troops git@github.com:xen-troops/meta-renesas.git
 git fetch troops
 git stash
 git cherry-pick f806ddaf40c465378c483004359404a9f141dc98^..405b9b98e5831c16e6b968a2cddce0a5fdac1856
 git stash apply
* Consider changes to r8a77951-salvator-xs-xen.dts introduced by one of the previous patches in case you want system configuration changes (Dom0 command line, Xen command line, IPMMU settings, etc)
* Skip build steps #5, #6 and continue from build step #7 till the end
* In build step #8 change the path to local.conf file:
 cp $WORK/meta-renesas/meta-rcar-gen3/docs/sample/conf/salvator-x/poky-gcc/bsp/*.conf ./conf/
 cd $WORK/build
 cp conf/local-wayland.conf conf/local.conf
= Build Xen =
== Get Xen sources ==
There is no need in additional patches here to bring up the system with Xen and Dom0, the mainline Xen works out of the box. Clone sources from the upstream and checkout on commit 73c932d0ea43ddf904db9429811788480c4cb816 “tools/libxc: use uint32_t for pirq in xc_domain_irq_permission”. Please note, most likely the more recent version of Xen will work as well, but the proposed commit is known to work:
 git clone git://xenbits.xenproject.org/xen.git
 cd xen
 git checkout 73c932d0ea43ddf904db9429811788480c4cb816 -b v4.16_xen
== Configure Xen ==
 export CROSS_COMPILE=<path_to_gcc>
 XEN_TARGET_ARCH=arm64 ./configure
 cd xen
 make menuconfig XEN_TARGET_ARCH=arm64
Enable IPMMU-VMSA support. Please note, the DMA devices in Dom0 will work even with IPMMU being disabled, as Dom0 is a direct mapped domain on Arm for now, therefore the IPMMU here is for protection purpose. But, if you are going to launch other domains (which are not direct mapped) and assign DMA devices to them, the IPMMU must be enabled to perform proper address translations for these devices to work:
 Device Drivers --->
 [*] Renesas IPMMU-VMSA found in R-Car Gen3 SoCs
Enable earlyprintk support (if needed):
 Debugging Options --->
 [*] Early printk (Early printk via SCIF UART) --->
 (X) Early printk via SCIF UART
 (0xe6e88000) Early printk, physical base address of debug UART
 Early printk UART SCIF interface version (default SCIF UART interface) --->
 (X) default SCIF UART interface
== Make Xen ==
 cd ..
 make xen XEN_TARGET_ARCH=arm64
 mkimage -A arm64 -C none -T kernel -a 0x78080000 -e 0x78080000 -n "XEN" -d xen/xen xen-uImage
 make -C tools/flask/policy
Copy resulting images to TFTP directory (prepared beforehand) for the future use:
 sudo cp xen-uImage <path_to_tftp_dir>
 sudo cp tools/flask/policy/xenpolicy* <path_to_tftp_dir>/xenpolicy
= Run the system =
* Setup TFTP and NFS (including the U-Boot environment) according to the instruction. Please note, the default expected NFS settings are the following: "nfsroot=192.168.1.100:/srv/salvator-x,vers=3 ip=dhcp". Also make sure that TFTP directory contains all target images, in addition to "r8a77951-salvator-xs-xen.dtb" and "Image" two new images need to be loaded: "xen-uImage" and "xenpolicy". 
* Update bootloaders on the board. It is crucial to update ArmTF in order to get Xen start in Non-secure EL2 mode. Follow your board manual and Yocto BSP documentation to find out the bootloaders update procedure.
* For the automatic boot add the following commands from the U-Boot CLI:
 setenv bootcmd run bootcmd_tftp
 setenv bootcmd_tftp 'run tftp_xen_load; run tftp_dtb_load; run tftp_kernel_load; run tftp_xenpolicy_load; bootm 0x48080000 - 0x48000000'
 setenv tftp_dtb_load tftp 0x48000000 r8a77951-salvator-xs-xen.dtb
 setenv tftp_kernel_load tftp 0x8a000000 Image
 setenv tftp_xen_load tftp 0x48080000 xen-uImage
 setenv tftp_xenpolicy_load tftp 0x8c000000 xenpolicy
 setenv bootargs
 saveenv
After rebooting the board, the system with Xen and Dom0 will start automatically if everything was done correctly.