KVM: ARM: Hypervisor inititalization

Sets up the required registers to run code in HYP-mode from the kernel.

By setting the HVBAR the kernel can execute code in Hyp-mode with
the MMU disabled. The HVBAR initially points to initialization code,
which initializes other Hyp-mode registers and enables the MMU
for Hyp-mode. Afterwards, the HVBAR is changed to point to KVM
Hyp vectors used to catch guest faults and to switch to Hyp mode
to perform a world-switch into a KVM guest.

Also provides memory mapping code to map required code pages, data structures,
and I/O regions  accessed in Hyp mode at the same virtual address as the host
kernel virtual addresses, but which conforms to the architectural requirements
for translations in Hyp mode. This interface is added in arch/arm/kvm/arm_mmu.c
and is comprised of:
 - create_hyp_mappings(from, to);
 - create_hyp_io_mappings(from, to, phys_addr);
 - free_hyp_pmds();

Note: The initialization mechanism currently relies on an SMC #0 call
to the secure monitor, which was merely a fast way of getting to the
hypervisor. We are working on supporting Hyp mode boot of the kernel
and control of Hyp mode through a local kernel mechanism.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Memory virtualization setup

This commit introduces the framework for guest memory management
through the use of 2nd stage translation. Each VM has a pointer
to a level-1 table (the pgd field in struct kvm_arch) which is
used for the 2nd stage translations. Entries are added when handling
guest faults (later patch) and the table itself can be allocated and
freed through the following functions implemented in
arch/arm/kvm/arm_mmu.c:
 - kvm_alloc_stage2_pgd(struct kvm *kvm);
 - kvm_free_stage2_pgd(struct kvm *kvm);

Introduces new ARM-specific kernel memory types, PAGE_KVM_GUEST and
pgprot_guest variables used to map 2nd stage memory for KVM guests.

Each entry in TLBs and caches are tagged with a VMID identifier in
addition to ASIDs. The VMIDs are assigned consecutively to VMs in the
order that VMs are executed, and caches and tlbs are invalidated when
the VMID space has been used to allow for more than 255 simultaenously
running guests.

The 2nd stage pgd is allocated in kvm_arch_init_vm(). The table is
freed in kvm_arch_destroy_vm(). Both functions are called from the main
KVM code.

We pre-allocate page table memory to be able to synchronize using a
spinlock and be called under rcu_read_lock from the MMU notifiers.  We
steal the mmu_memory_cache implementation from x86 and adapt for our
specific usage.

We support MMU notifiers (thanks to Marc Zyngier) through
kvm_unmap_hva and kvm_set_spte_hva.

Finally, define kvm_phys_addr_ioremap() to map a device at a guest IPA,
which is used by VGIC support to map the virtual CPU interface registers
to the guest. This support is added by Marc Zyngier.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Inject IRQs and FIQs from userspace

Userspace can inject IRQs and FIQs through the KVM_IRQ_LINE VM ioctl.
This ioctl is used since the sematics are in fact two lines that can be
either raised or lowered on the VCPU - the IRQ and FIQ lines.

KVM needs to know which VCPU it must operate on and whether the FIQ or
IRQ line is raised/lowered. Hence both pieces of information is packed
in the kvm_irq_level->irq field. The irq fild value will be:
  IRQ: vcpu_index << 1
  FIQ: (vcpu_index << 1) | 1

This is documented in Documentation/kvm/api.txt.

The effect of the ioctl is simply to simply raise/lower the
corresponding irq_line field on the VCPU struct, which will cause the
world-switch code to raise/lower virtual interrupts when running the
guest on next switch. The wait_for_interrupt flag is also cleared for
raised IRQs or FIQs causing an idle VCPU to become active again. CPUs
in guest mode are kicked to make sure they refresh their interrupt status.

Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: World-switch implementation

Provides complete world-switch implementation to switch to other guests
running in non-secure modes. Includes Hyp exception handlers that
capture necessary exception information and stores the information on
the VCPU and KVM structures.

The following Hyp-ABI is also documented in the code:

Hyp-ABI: Switching from host kernel to Hyp-mode:
   Switching to Hyp mode is done through a simple HVC instructions. The
   exception vector code will check that the HVC comes from VMID==0 and if
   so will store the necessary state on the Hyp stack, which will look like
   this (growing downwards, see the hyp_hvc handler):
     ...
     stack_page + 4: spsr (Host-SVC cpsr)
     stack_page    : lr_usr
     --------------: stack bottom

Hyp-ABI: Switching from Hyp-mode to host kernel SVC mode:
   When returning from Hyp mode to SVC mode, another HVC instruction is
   executed from Hyp mode, which is taken in the hyp_svc handler. The
   bottom of the Hyp is derived from the Hyp stack pointer (only a single
   page aligned stack is used per CPU) and the initial SVC registers are
   used to restore the host state.

Otherwise, the world-switch is pretty straight-forward. All state that
can be modified by the guest is first backed up on the Hyp stack and the
VCPU values is loaded onto the hardware. State, which is not loaded, but
theoretically modifiable by the guest is protected through the
virtualiation features to generate a trap and cause software emulation.
Upon guest returns, all state is restored from hardware onto the VCPU
struct and the original state is restored from the Hyp-stack onto the
hardware.

SMP support using the VMPIDR calculated on the basis of the host MPIDR
and overriding the low bits with KVM vcpu_id contributed by Marc Zyngier.

Reuse of VMIDs has been implemented by Antonios Motakis and adapated from
a separate patch into the appropriate patches introducing the
functionality. Note that the VMIDs are stored per VM as required by the ARM
architecture reference manual.

To support VFP/NEON we trap those instructions using the HPCTR. When
we trap, we switch the FPU.  After a guest exit, the VFP state is
returned to the host.  When disabling access to floating point
instructions, we also mask FPEXC_EN in order to avoid the guest
receiving Undefined instruction exceptions before we have a chance to
switch back the floating point state.  We are reusing vfp_hard_struct,
so we depend on VFPv3 being enabled in the host kernel, if not, we still
trap cp10 and cp11 in order to inject an undefined instruction exception
whenever the guest tries to use VFP/NEON. VFP/NEON developed by
Antionios Motakis and Rusty Russell.

Signed-off-by: Rusty Russell <rusty.russell@linaro.org>
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Emulation framework and CP15 emulation

Adds a new important function in the main KVM/ARM code called
handle_exit() which is called from kvm_arch_vcpu_ioctl_run() on returns
from guest execution. This function examines the Hyp-Syndrome-Register
(HSR), which contains information telling KVM what caused the exit from
the guest.

Some of the reasons for an exit are CP15 accesses, which are
not allowed from the guest and this commit handles these exits by
emulating the intended operation in software and skip the guest
instruction.

Minor notes about the coproc register reset:
1) We reserve a value of 0 as an invalid cp15 offset, to catch bugs in our
   table, at cost of 4 bytes per vcpu.

2) Added comments on the table indicating how we handle each register, for
   simplicity of understanding.

Signed-off-by: Rusty Russell <rusty.russell@linaro.org>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: User space API for getting/setting co-proc registers

The following three ioctls are implemented:
 -  KVM_VCPU_GET_MSR_INDEX_LIST
 -  KVM_GET_MSRS
 -  KVM_SET_MSRS

Now we have a table for all the cp15 registers, we can drive a generic
API.  x86 already has one for sparse-numbered registers, so we simply
reproduce that.  The only difference is that our KVM_GET_MSR_INDEX_LIST
is a per-vcpu ioctl; we can't know the MSRs until we know the cpu type.
Thus we add KVM_VCPU_GET_MSR_INDEX_LIST.

The numbering for the indices for coprocesors is simple, if userspace
cares (it might not for simple save and restore): the upper 16 bits
are the coprocessor number.  If it's > 15, it's something else, for
future expansion.

Bit 15 indicates a 64-bit register.  For 64 bit registers the bottom 4
bits are CRm, the next 4 are opc1 (just like the MCRR/MRCC instruction
encoding).  For 32 bit registers, the bottom 4 bits are CRm, the next
3 are opc2, the next 4 CRn, and the next 3 opc1 (the same order as the
MRC/MCR instruction encoding, but not the same bit positions).

64-bit coprocessor register:
       ...|19 18 17 16|15|14 13 12 11 10  9  8| 7  6  5  4 |3  2  1  0|
  ...0  0 |  cp num   | 1| 0  0  0  0  0  0  0|   opc1     |   CRm    |

32-bit coprocessor register:
       ...|19 18 17 16|15|14|13 12 11|10  9  8  7 |6  5  4 |3  2  1  0|
  ...0  0 |  cp num   | 0| 0|  opc1  |    CRn     | opc2   |   CRm    |

Non-coprocessor register:

   | 32 31 30 29 28 27 26 25 24 23 22 21 20|19 18 17 16 15 ...
   |     < some non-zero value >           | ...

For futureproofing, we need to tell QEMU about the CP15 registers the
host lets the guest access.

It will need this information to restore a current guest on a future
CPU or perhaps a future KVM which allow some of these to be changed.

We use a separate table for these, as they're only for the userspace API.

Signed-off-by: Rusty Russell <rusty.russell@linaro.org>
KVM: ARM: Handle guest faults in KVM

Handles the guest faults in KVM by mapping in corresponding user pages
in the 2nd stage page tables.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Handle I/O aborts

When the guest accesses I/O memory this will create data abort
exceptions and they are handled by decoding the HSR information
(physical address, read/write, length, register) and forwarding reads
and writes to QEMU which performs the device emulation.

Certain classes of load/store operations do not support the syndrome
information provided in the HSR and we therefore must be able to fetch
the offending instruction from guest memory and decode it manually.

This requires changing the general flow somewhat since new calls to run
the VCPU must check if there's a pending MMIO load and perform the write
after userspace has made the data available.

Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Guest wait-for-interrupts (WFI) support

When the guest executes a WFI instruction the operation is trapped to
KVM, which emulates the instruction in software. There is no correlation
between a guest executing a WFI instruction and actually putting the
hardware into a low-power mode, since a KVM guest is essentially a
process and the WFI instruction can be seen as 'sleep' call from this
process. Therefore, we block the vcpu when the guest excecutes a wfi
instruction and the IRQ or FIQ lines are not raised.

When an interrupt comes in through KVM_IRQ_LINE (see previous patch) we
signal the VCPU thread and unflag the VCPU to no longer wait for
interrupts.

Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: A single hva can map multiple gpas

Handle mmu notifier ops for every such mapping.

Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Clarify comment on handle_exit.

If I'm reading the code right, > 0 loops around, < 0 exits with an error,
and 0 returns to qemu.

Signed-off-by: Rusty Russell <rusty.russell@linaro.org>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: __kvm_va_to_pa()

Create a helper to do the guest va->pa mapping.  Then take the code out of the
exit path, and use normal C.

copy_from_guest_va() is a bit more generic than it needs to be, but
not much.

Signed-off-by: Rusty Russell <rusty.russell@linaro.org>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Let guest instruction mapping fail.

There is a race (SMP guest and host only) where the instruction we
faulted on is unmapped before we get to reading it.  In this (very
unlikely!) case, we should simply return to the guest.  It will
probably do the right thing next time.

Signed-off-by: Rusty Russell <rusty.russell@linaro.org>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Atomically read guest instruction,

Ben Herrenschmidt pointed out a nasty race:
(1) Guest complicated mmio instruction traps.
(2) The hardware doesn't tell us enough, so we need to read the actual
    instruction which was being exectuted.
(3) KVM maps the instruction virtual address to a physical address.
(4) The guest (SMP) swaps out that page, and fills it with something else.
(5) We read the physical address, but now that's the wrong thing.

This is so rare that we can stop all the vcpus from running when it
happens (if no VCPUS are running, this just means getting and
releasing a spinlock).

Signed-off-by: Rusty Russell <rusty.russell@linaro.org>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: 16-bit Thumb instructions shouldn't generate error

When copying guest instructions on an 'invalid' IO abort, we don't want
to just greedily copy 32 bits, as a 16-bit thumb instruction at the end
of a page may exist such that the following page is not mapped in the
guest and our attempt to translate the VA->IPA of that page will just
continously fail.  No, instead we do this madness twice, checking the
length of the instruciton in between.

Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Inject data/prefetch aborts in guest

Inject these aborts if the guest tries to do something really weird.
It's nicer to tell the guest that there was an abort than just
forcefully kill the whole thing.

For now we always tell the guest it was a debug generated abort, but we
can make this scheme more flexible in the future.

Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Cleanup and fix load/store instruction decoding

Split the kvm_emulate_mmio_ls into a separate decoding function
and also supports emulated mmio writes now.

Use an in-kernel mmio structure to prepare an mmio operation which can
later be copied to the user-mapped run struct.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Support Thumb MMIO instruction decoding

We only support instruction decoding for valid reasonable MMIO operations
where trapping them do not provide sufficient information in the HSR (no
16-bit Thumb instructions provide register writeback that we care about).

The following instruciton types are NOT supported for MMIO operations
despite the HSR not containing decode info:
 - any Load/Store multiple
 - any load/store exclusive
 - any load/store dual
 - anything with the PC as the dest register

Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Inject data abort on weird MMIO instrucitons

Since there are MMIO instructions that we just don't want to emulate
because the guest would be crazy to use them, we want to tell the guest
that this is what happened and not kill the VM aprubtly.

Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
ARM: KVM: simplify stack layout in HYP mode

On world switch, we save the pointer to the current vcpu on the top
of the stack. This tends to complicate the exit path a bit as we then
need to dig into the stack to restore it (we've already pushed a number
of registers on top of it at that time), not to mention error prone.

A obvious way to simplify this is to use the HTPIDR register, which is an
architected method to store per-CPU data in HYP mode, and this is
exactly what this patch does.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>