Xen ARM with Virtualization Extensions/Allwinner

== Hardware ==
This wiki talks about running Xen on systems with an Allwinner SoC (often called sunxi) with virtualization capable cores (ARM Cortex-A7 or Cortex-A53).
This is well tested and supported on A20 SoCs (Cortex-A7 dual core, sun7i). Since v4.10-rc1 there is support for more SoCs, including arm64 capable chips (A64, H5).
The following boards have been tested:
* A20
** Cubieboard2 (note that the original Cubieboard does not support virtualisation)
*** Cubieboard2 with Xen: step-by-step guide
*** Cubieboard 2 with Arch Linux ARM as Dom0 and DomU
** Cubietruck (AKA Cubieboard 3)
* H2+/H3
** OrangePi Zero
* A64
** Pine64
* H5
** OrangePi PC 2
== Bootloader ==
As described in Firmware Requirements, Xen needs to be booted in non-secure HYP mode (EL2 on ARM64).
Since at least v2014.11 U-Boot supports all the required technologies to support Xen: it supports most boards using an Allwinner SoC, boots the kernel (Xen in our case) in HYP mode and provides an PSCI interface for SMP bringup. There are no special options needed to get a Xen-capable U-Boot, just configure it for your board and build normally. Normal firmware builds (for instance as provided by distributions) should just work.
== Linux ==
=== Upstream Support ===
Support for the Allwinner processors in Linux is mainly a community effort, see the Linux Sunxi Wiki. Support for Allwinner SoCs in the mainline kernel is constantly improving, most SoCs are well supported for a headless setup (serial, MMC, SATA, USB, Ethernet). Check the latest status.
=== Building ===
None of the kernel provided defconfigs for ARM(32) include the required Xen options. To get a Xen capable kernel, use either multi_v7_defconfig or sunxi_defconfig as a base or use your distribution provided config file.
From here you will want to enable the various Xen options (CONFIG_XEN, frontend and backend drivers etc, see Mainline_Linux_Kernel_Configs for more general details).
Then:
 make zImage
 make dtbs
The final kernel binary will be in `arch/arm/boot/zImage` and the device tree will be located in `arch/arm/boot/dts` e.g. sun7i-a20-cubieboard2.dtb, sun7i-a20-cubietruck.dtb etc.
The arm64 defconfig includes the Xen options since Linux v4.5, so most kernels should work out of the box.
The same kernel image should work for both dom0 and domU as well as for bare-metal boots.
== Building Xen ==
The latest xen should work. See Build Xen on ARM and Building Xen and Linux Dom0 for building xen and generating the xen-uImage.
Use the following build command for enable early printk for a20:
 make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-linux-gnueabihf- CONFIG_EARLY_PRINTK=sun7i
== Booting Xen and Dom0 ==
=== Command lines ===
For Xen the correct command line for console on the first serial port is `dtuart=/soc@01c00000/serial@01c28000`. For other serial ports use the correct name/address from the device tree.
For dom0 an additional `clk_ignore_unused` option must be included in the dom0 kernel command line.
=== Load addresses ===
Due to Xen ARM TODO/Domain 0 memory limitation due to 1:1 mapping it is recommended to place the initial binary blobs towards the top of RAM, leaving 2MB free for Xen to relocate into.
The following table shows suggested load addresses for 1GB (e.g cubieboard2) and 2GB (e.g. cubietruck) systems. These can be used with the boot.scr boot script described in the next section.
Binary 	1GB systems	2GB systems	Size	boot.scr variable name
Top of RAM 	0x80000000 	0xc0000000 	 	
Xen relocation target address 	0x7fe00000 	0xbfe00000 	2MB 	
Dom0 kernel (Linux vmlinuz) 	0x7f600000 	0xbf600000 	8MB 	kernel_addr_r
Dom0 initial ramdisk (optional)	0x7ee00000 	0xbee00000 	8MB 	ramdisk_addr_r
Device tree blob 	0x7ec00000 	0xbec00000 	2MB 	fdt_addr
Xen initial load address 	0x7ea00000 	0xbea00000 	2MB 	xen_addr_r
This assumes that the kernel is <8MB, if it is larger then shift everything down in 2MB increments. Likewise for the initial ramdisk.
=== Boot script ===
To boot Xen and Dom0, you can use USB boot or sd boot. Some runtime manipulation of the DTB is needed, but the basic outline is the same in each case (only the method to load the binary differs).
 # SUNXI Xen Boot Script
 
 # Addresses suitable for 1GB system, adjust as appropriate for a 2GB system.
 # Top of RAM: 0x80000000
 # Xen relocate addr 0x7fe00000
 setenv kernel_addr_r 0x7f600000 # 8M
 setenv ramdisk_addr_r 0x7ee00000 # 8M
 setenv fdt_addr 0x7ec00000 # 2M
 setenv xen_addr_r 0x7ea00000 # 2M
 
 setenv fdt_high 0xffffffff # Load fdt in place instead of relocating
 
 # Load xen/xen to ${xen_addr_r}. e.g. tftp, fatload or ext2load to ${xen_addr_r}.
 # see the following sections for details of booting from various devices.
 
 setenv bootargs "console=dtuart dtuart=/soc@01c00000/serial@01c28000 dom0_mem=128M"
 
 # Load appropriate .dtb file to ${fdt_addr} e.g. tftp, fatload or ext2load to ${fdt_addr}.
 # see the following sections for details of booting from various devices.
 
 fdt addr ${fdt_addr} 0x40000
 
 fdt resize
 
 fdt chosen
 
 fdt set /chosen \#address-cells <1>
 fdt set /chosen \#size-cells <1>
 
 # Load Linux arch/arm/boot/zImage to ${kernel_addr_r}. e.g. tftp, fatload or ext2load to ${kernel_addr_r}.
 # see the following sections for details of booting from various devices.
 
 fdt mknod /chosen module@0
 fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
 fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
 fdt set /chosen/module@0 bootargs "console=hvc0 ro root=/dev/sda1 clk_ignore_unused"
 
 bootz ${xen_addr_r} - ${fdt_addr}
The easiest approach is to create a file (boot.xen) with this and then convert this into a u-boot boot.scr with:
 mkimage -A arm -T script -d boot.xen boot.scr.
This can then be loaded to address 0x41000000 and then launched with:
 source 0x41000000
=== Booting via PXE ===
To boot use the boot.scr above loading images with:
 tftp ${address} /path/to/image
For example:
 # Load xen/xen to ${xen_addr_r}
 tftp ${xen_addr_r} /cubieboard/xen
 
 # Load appropriate .dtb file to ${fdt_addr}
 tftp ${fdt_addr} /cubieboard/sun7i-a20-cubieboard2.dtb
 
 # Load Linux arch/arm/boot/zImage to ${kernel_addr_r}
 tftp ${kernel_addr_r} /cubieboard/vmlinuz
To boot this set bootcmd:
 setenv bootcmd setenv autoload no\;dhcp\;tftp 0x41000000 /boot.scr;source 0x41000000
 saveenv
 boot
=== Booting via USBBoot(FEL) ===
reference: http://linux-sunxi.org/FEL/USBBoot
Arrange for all of the images to be loaded at the appropriate addresses via the FEL utility and then load a suitable boot.scr to launch them.
=== Booting directly on the SD card ===
reference Bootable SD Card for how to make a bootable sdcard.
Copy xen, zImage, the dtb and boot.scr to the root directory of you SD card.
To boot use the boot.scr above loading images with:
 fatload mmc 0 ${address} /path/to/image
For example:
 # Load xen/xen to ${xen_addr_r}
 fatload mmc 0 ${xen_addr_r} /xen
 
 # Load appropriate .dtb file to ${fdt_addr}
 fatload mmc 0 ${fdt_addr} /sun7i-a20-cubieboard2.dtb
 
 # Load Linux arch/arm/boot/zImage to ${kernel_addr_r}
 fatload mmc 0 ${kernel_addr_r} /vmlinuz
To boot set the bootcmd:
 setenv bootcmd "fatload mmc 0 0x41000000 boot.scr; source 0x41000000"
 savenv
 boot
This assume the SD card has only 1 partition with fat filesystem. Use e.g. ext2load etc if this is not the case.
=== Mount NFS after initrd boot ===
Using a distrbution is easily to compile xen tools.
Both opensuse arm 12.3 rootfs and Debian have been used by developers. Other distribution which support armhf is ok too.
== Create domU ==
;setup the harddisk
 losetup /dev/loop0 /root/domU/rootfs.ext3 
;domU config example
 > cat domU_test
 kernel = "/root/domU/zImage
 memory = "128" 
 name = "domU" 
 vcpus = 1 
 serial="pty"
 disk = [ 'phy:/dev/loop0,xvda,w' ]
 vif=[ 'mac=00:16:3e:56:af:69,bridge=virbr0,type=netfront', ]
;create
 xl create domU_test
or enable all the debug message
 xl -vvv create -d domU_testa
;connect to console
xl console domU_test
== DomU kernel debug ==
Xen supports a variety of debug methods. You can use xl debug-keys for dumping information. Use `xen_raw_console_write` for output DomU kernel output.
`gdbsx` is not work for arm right now.
 diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
 index b4e8500..060c163 100644
 --- a/kernel/printk/printk.c
 +++ b/kernel/printk/printk.c
 @@ -1488,6 +1488,7 @@ static size_t cont_print_text(char *text, size_t size)
 return textlen;
 }
 
 +#include <xen/hvc-console.h>
 asmlinkage int vprintk_emit(int facility, int level,
 const char *dict, size_t dictlen,
 const char *fmt, va_list args)
 @@ -1546,6 +1547,7 @@ asmlinkage int vprintk_emit(int facility, int level,
 * prefix which might be passed-in as a parameter.
 */
 text_len = vscnprintf(text, sizeof(textbuf), fmt, args);
 + xen_raw_console_write(text);
 
 /* mark and strip a trailing newline */
 if (text_len && text[text_len-1] == '\n') {
== Resources ==
allwinner sunxi resources
Category:XenARM
Category:Developers
Category:Xen 4.4
Category:Xen 4.5