ARM: add mem_type prot_pte accessor

The KVM hypervisor mmu code requires access to the mem_type prot_pte
field when setting up page tables pointing to a device. Unfortunately,
the mem_type structure is opaque.

Add an accessor (get_mem_type_prot_pte()) to retrieve the prot_pte
value.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
ARM: Add config option ARM_VIRT_EXT

Select this option for ARM processors equipped with hardware
Virtualization Extensions.

Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
ARM: Section based HYP idmap

Add a HYP pgd to the core code (so it can benefit all Linux
hypervisors).

Populate this pgd with an identity mapping of the code contained
in the .hyp.idmap.text section

Offer a method to drop the this identity mapping through
hyp_idmap_teardown and re-create it through hyp_idmap_setup.

Make all the above depend on CONFIG_ARM_VIRT_EXT

Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
ARM: Expose PMNC bitfields for KVM use

We want some of these for use in KVM, so pull them out of
arch/arm/kernel/perf_event_v7.c into their own asm/perf_bits.h.

Signed-off-by: Rusty Russell <rusty.russell@linaro.org>
KVM: ARM: Initial skeleton to compile KVM support

Targets KVM support for Cortex A-15 processors.

Contains all the framework components, make files, header files and some
tracing functionality.

Only supported core is Cortex-A15 for now.

Contains minor reset hook driven from kvm_vcpu_set_target, which will
eventually be a custom ARM ioctl to set the core we are emulating.

Most functionality is in arch/arm/kvm/* or arch/arm/include/asm/kvm_*.h.

“Nothing to see here. Move along, move along..."

Signed-off-by: Rusty Russell <rusty.russell@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Hypervisor inititalization

Sets up the required registers to run code in HYP-mode from the kernel.

By setting the HVBAR the kernel can execute code in Hyp-mode with
the MMU disabled. The HVBAR initially points to initialization code,
which initializes other Hyp-mode registers and enables the MMU
for Hyp-mode. Afterwards, the HVBAR is changed to point to KVM
Hyp vectors used to catch guest faults and to switch to Hyp mode
to perform a world-switch into a KVM guest.

Also provides memory mapping code to map required code pages, data structures,
and I/O regions  accessed in Hyp mode at the same virtual address as the host
kernel virtual addresses, but which conforms to the architectural requirements
for translations in Hyp mode. This interface is added in arch/arm/kvm/arm_mmu.c
and is comprised of:
 - create_hyp_mappings(from, to);
 - create_hyp_io_mappings(from, to, phys_addr);
 - free_hyp_pmds();

Note: The initialization mechanism currently relies on an SMC #0 call
to the secure monitor, which was merely a fast way of getting to the
hypervisor. We are working on supporting Hyp mode boot of the kernel
and control of Hyp mode through a local kernel mechanism.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Memory virtualization setup

This commit introduces the framework for guest memory management
through the use of 2nd stage translation. Each VM has a pointer
to a level-1 table (the pgd field in struct kvm_arch) which is
used for the 2nd stage translations. Entries are added when handling
guest faults (later patch) and the table itself can be allocated and
freed through the following functions implemented in
arch/arm/kvm/arm_mmu.c:
 - kvm_alloc_stage2_pgd(struct kvm *kvm);
 - kvm_free_stage2_pgd(struct kvm *kvm);

Introduces new ARM-specific kernel memory types, PAGE_KVM_GUEST and
pgprot_guest variables used to map 2nd stage memory for KVM guests.

Each entry in TLBs and caches are tagged with a VMID identifier in
addition to ASIDs. The VMIDs are assigned consecutively to VMs in the
order that VMs are executed, and caches and tlbs are invalidated when
the VMID space has been used to allow for more than 255 simultaenously
running guests.

The 2nd stage pgd is allocated in kvm_arch_init_vm(). The table is
freed in kvm_arch_destroy_vm(). Both functions are called from the main
KVM code.

We pre-allocate page table memory to be able to synchronize using a
spinlock and be called under rcu_read_lock from the MMU notifiers.  We
steal the mmu_memory_cache implementation from x86 and adapt for our
specific usage.

We support MMU notifiers (thanks to Marc Zyngier) through
kvm_unmap_hva and kvm_set_spte_hva.

Finally, define kvm_phys_addr_ioremap() to map a device at a guest IPA,
which is used by VGIC support to map the virtual CPU interface registers
to the guest. This support is added by Marc Zyngier.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Inject IRQs and FIQs from userspace

Userspace can inject IRQs and FIQs through the KVM_IRQ_LINE VM ioctl.
This ioctl is used since the sematics are in fact two lines that can be
either raised or lowered on the VCPU - the IRQ and FIQ lines.

KVM needs to know which VCPU it must operate on and whether the FIQ or
IRQ line is raised/lowered. Hence both pieces of information is packed
in the kvm_irq_level->irq field. The irq fild value will be:
  IRQ: vcpu_index << 1
  FIQ: (vcpu_index << 1) | 1

This is documented in Documentation/kvm/api.txt.

The effect of the ioctl is simply to simply raise/lower the
corresponding irq_line field on the VCPU struct, which will cause the
world-switch code to raise/lower virtual interrupts when running the
guest on next switch. The wait_for_interrupt flag is also cleared for
raised IRQs or FIQs causing an idle VCPU to become active again. CPUs
in guest mode are kicked to make sure they refresh their interrupt status.

Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: World-switch implementation

Provides complete world-switch implementation to switch to other guests
running in non-secure modes. Includes Hyp exception handlers that
capture necessary exception information and stores the information on
the VCPU and KVM structures.

The following Hyp-ABI is also documented in the code:

Hyp-ABI: Switching from host kernel to Hyp-mode:
   Switching to Hyp mode is done through a simple HVC instructions. The
   exception vector code will check that the HVC comes from VMID==0 and if
   so will store the necessary state on the Hyp stack, which will look like
   this (growing downwards, see the hyp_hvc handler):
     ...
     stack_page + 4: spsr (Host-SVC cpsr)
     stack_page    : lr_usr
     --------------: stack bottom

Hyp-ABI: Switching from Hyp-mode to host kernel SVC mode:
   When returning from Hyp mode to SVC mode, another HVC instruction is
   executed from Hyp mode, which is taken in the hyp_svc handler. The
   bottom of the Hyp is derived from the Hyp stack pointer (only a single
   page aligned stack is used per CPU) and the initial SVC registers are
   used to restore the host state.

Otherwise, the world-switch is pretty straight-forward. All state that
can be modified by the guest is first backed up on the Hyp stack and the
VCPU values is loaded onto the hardware. State, which is not loaded, but
theoretically modifiable by the guest is protected through the
virtualiation features to generate a trap and cause software emulation.
Upon guest returns, all state is restored from hardware onto the VCPU
struct and the original state is restored from the Hyp-stack onto the
hardware.

SMP support using the VMPIDR calculated on the basis of the host MPIDR
and overriding the low bits with KVM vcpu_id contributed by Marc Zyngier.

Reuse of VMIDs has been implemented by Antonios Motakis and adapated from
a separate patch into the appropriate patches introducing the
functionality. Note that the VMIDs are stored per VM as required by the ARM
architecture reference manual.

To support VFP/NEON we trap those instructions using the HPCTR. When
we trap, we switch the FPU.  After a guest exit, the VFP state is
returned to the host.  When disabling access to floating point
instructions, we also mask FPEXC_EN in order to avoid the guest
receiving Undefined instruction exceptions before we have a chance to
switch back the floating point state.  We are reusing vfp_hard_struct,
so we depend on VFPv3 being enabled in the host kernel, if not, we still
trap cp10 and cp11 in order to inject an undefined instruction exception
whenever the guest tries to use VFP/NEON. VFP/NEON developed by
Antionios Motakis and Rusty Russell.

Signed-off-by: Rusty Russell <rusty.russell@linaro.org>
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Emulation framework and CP15 emulation

Adds a new important function in the main KVM/ARM code called
handle_exit() which is called from kvm_arch_vcpu_ioctl_run() on returns
from guest execution. This function examines the Hyp-Syndrome-Register
(HSR), which contains information telling KVM what caused the exit from
the guest.

Some of the reasons for an exit are CP15 accesses, which are
not allowed from the guest and this commit handles these exits by
emulating the intended operation in software and skip the guest
instruction.

Minor notes about the coproc register reset:
1) We reserve a value of 0 as an invalid cp15 offset, to catch bugs in our
   table, at cost of 4 bytes per vcpu.

2) Added comments on the table indicating how we handle each register, for
   simplicity of understanding.


Signed-off-by: Rusty Russell <rusty.russell@linaro.org>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: User space API for getting/setting co-proc registers

The following three ioctls are implemented:
 -  KVM_VCPU_GET_MSR_INDEX_LIST
 -  KVM_GET_MSRS
 -  KVM_SET_MSRS

Now we have a table for all the cp15 registers, we can drive a generic
API.  x86 already has one for sparse-numbered registers, so we simply
reproduce that.  The only difference is that our KVM_GET_MSR_INDEX_LIST
is a per-vcpu ioctl; we can't know the MSRs until we know the cpu type.
Thus we add KVM_VCPU_GET_MSR_INDEX_LIST.

The numbering for the indices for coprocesors is simple, if userspace
cares (it might not for simple save and restore): the upper 16 bits
are the coprocessor number.  If it's > 15, it's something else, for
future expansion.

Bit 15 indicates a 64-bit register.  For 64 bit registers the bottom 4
bits are CRm, the next 4 are opc1 (just like the MCRR/MRCC instruction
encoding).  For 32 bit registers, the bottom 4 bits are CRm, the next
3 are opc2, the next 4 CRn, and the next 3 opc1 (the same order as the
MRC/MCR instruction encoding, but not the same bit positions).

64-bit coprocessor register:
       ...|19 18 17 16|15|14 13 12 11 10  9  8| 7  6  5  4 |3  2  1  0|
  ...0  0 |  cp num   | 1| 0  0  0  0  0  0  0|   opc1     |   CRm    |

32-bit coprocessor register:
       ...|19 18 17 16|15|14|13 12 11|10  9  8  7 |6  5  4 |3  2  1  0|
  ...0  0 |  cp num   | 0| 0|  opc1  |    CRn     | opc2   |   CRm    |

Non-coprocessor register:

   | 32 31 30 29 28 27 26 25 24 23 22 21 20|19 18 17 16 15 ...
   |     < some non-zero value >           | ...


For futureproofing, we need to tell QEMU about the CP15 registers the
host lets the guest access.

It will need this information to restore a current guest on a future
CPU or perhaps a future KVM which allow some of these to be changed.

We use a separate table for these, as they're only for the userspace API.

Signed-off-by: Rusty Russell <rusty.russell@linaro.org>
KVM: ARM: Handle guest faults in KVM

Handles the guest faults in KVM by mapping in corresponding user pages
in the 2nd stage page tables.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Handle I/O aborts

When the guest accesses I/O memory this will create data abort
exceptions and they are handled by decoding the HSR information
(physical address, read/write, length, register) and forwarding reads
and writes to QEMU which performs the device emulation.

Certain classes of load/store operations do not support the syndrome
information provided in the HSR and we therefore must be able to fetch
the offending instruction from guest memory and decode it manually.

This requires changing the general flow somewhat since new calls to run
the VCPU must check if there's a pending MMIO load and perform the write
after userspace has made the data available.

Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
KVM: ARM: Guest wait-for-interrupts (WFI) support

When the guest executes a WFI instruction the operation is trapped to
KVM, which emulates the instruction in software. There is no correlation
between a guest executing a WFI instruction and actually putting the
hardware into a low-power mode, since a KVM guest is essentially a
process and the WFI instruction can be seen as 'sleep' call from this
process. Therefore, we block the vcpu when the guest excecutes a wfi
instruction and the IRQ or FIQ lines are not raised.

When an interrupt comes in through KVM_IRQ_LINE (see previous patch) we
signal the VCPU thread and unflag the VCPU to no longer wait for
interrupts.

Signed-off-by: Christoffer Dall <c.dall@virtualopensystems.com>
ARM: KVM: Clear excusive monitor on world switch

If a VM exits in the middle of an exclusive operation (ldrex/strex),
it leaves the exclusive monitor in a state that can influence the host
(specially if they all run the same kernel or userspace, as all the
virtual addresses will be the same). Conversly, the host state can
influence the guest in the same way.

This has been observed on TC2 running two VMs with the same kernel as
the host, and seeing deadlocks (or lockdep moaning about locks being
already taken by another CPU).

The fix is to clear the exclusive monitor on both exception return
paths (which is consistent with what the kernel is doing for userspace)
so that the different worlds are properly isolated.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
ARM: KVM: Don't corrupt USR registers with kvm_arch_vcpu_ioctl_{gs}et_regs

r0/r1 are getting mixed up with r13/r14. Probably a copy/paste issue.

Reported-by: Andreas Sandberg <andreas.sandberg@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
ARM: KVM: remove CONFIG_ARM_VIRT_EXT

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
ARM: KVM: Change KVM/ARM dependencies

We used to have ARM_VIRT_EXT depending on ARM_LPAE, and KVM
depending on CPU_V7 && ARM_VIRT_EXT.

ARM_VIRT_EXT doesn't depend on ARM_LPAE anymore (no reason it
should, this is a KVM limitation), but now depends on CPU_v7 & MMU.

To preserve the dependencies, change KVM to depend on ARM_VIRT_EXT &
ARM_LPAE.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
ARM: KVM: Switch HVBAR setting to be HVC-based

Instead of relying on SMC calls to setup HVBAR, introduce a new
HVC-based API. This of course requires a hypervisor stub to be
installed already. At the moment, this is implemented as part of the
boot wrapper, but will be moved to the kernel itself when it actually
understands being booted in HYP mode.

The kernel now uses "hvc #0xff", with r0 containing the pointer to
the new vectors.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
ARM: KVM: make init.S Thumb compliant

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>