fix commit xen/arm: Add support for GICv3 for domU

The build of xen-4.5.0-rc2 fails if XSM_ENABLE=y due to an inconsistency
in commit fda1614 "xen/arm: Add support for GICv3 for domU" which uses
XEN_DOMCTL_configure_domain in xen/xsm/flask/hooks.c and
xen/xsm/flask/policy/access_vectors but XEN_DOMCTL_arm_configure_domain
elsewhere.

 	Michael Young
In fda1614 ("xen/arm: Add support for GICv3 for domU")
XEN_DOMCTL_configure_domain is used in xen/xsm/flask/hooks.c and
xen/xsm/flask/policy/access_vectors but XEN_DOMCTL_arm_configure_domain
is used elsewhere.

Signed-off-by: Michael Young <m.a.young@durham.ac.uk>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
sched_rt: sanity check input and serialization

Sanity check input params in rt_dom_cntl();
Serialize rt_dom_cntl() against the global lock.

Reported-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Meng Xu <mengxu@cis.upenn.edu>
Reviewed-by: Dario Faggioli <dario.faggioli@citrix.com>
Reviewed-by: George Dunlap <george.dunlap@eu.citrix.com>
Release-Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
sched_rt: serialize vcpu data access

Fix the following two issues in rtds scheduler:
1) The runq queue lock is not grabbed when rt_update_deadline is
called in rt_alloc_vdata function, which may cause race condition;
Solution: Move call to rt_update_deadline from _alloc to _insert;
Note: rt_alloc_vdata does not need grab the runq lock, because only one
cpu will allocate the rt_vcpu; before the rt_vcpu is inserted into the
runq, no more than one cpu operates on the rt_vcpu.

2) rt_vcpu_remove should grab the runq lock before remove the vcpu
from runq; otherwise, race condition may happen.
Solution: Add lock in rt_vcpu_remove().

Reported-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Meng Xu <mengxu@cis.upenn.edu>
Reviewed-by: Dario Faggioli <dario.faggioli@citrix.com>
Reviewed-by: George Dunlap <george.dunlap@eu.citrix.com>
Release-Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
dpci: move from an hvm_irq_dpci (and struct domain) to an hvm_dirq_dpci model

When an interrupt for an PCI (or PCIe) passthrough device is to be sent
to a guest, we find the appropiate 'hvm_dirq_dpci' structure for the
interrupt (PIRQ), set a bit (masked), and schedule an tasklet.

Then the 'hvm_dirq_assist' tasklet gets called with the 'struct domain'
from where it iterates over the the radix-tree of 'hvm_dirq_dpci' (from
zero to the number of PIRQs allocated) which are masked to the guest
and calls each 'hvm_pirq_assist'. If the PIRQ has a bit set (masked) it
figures out how to inject the PIRQ to the guest.

This is inefficient and not fair as:
 - We iterate starting at PIRQ 0 and up every time. That means the PCIe
   devices that have lower PIRQs get to be called first.
 - If we have many PCIe devices passed in with many PIRQs and if most
   of the time only the highest numbered PIRQ get an interrupt (as the
   initial ones are for control) we end up iterating over many PIRQs.

But we could do beter - the 'hvm_dirq_dpci' has the field for
'struct domain', which we can use instead of having to pass in the
'struct domain'.

As such this patch moves the tasklet to the 'struct hvm_dirq_dpci' and
sets the 'dom' field to the domain. We also double-check that the
'->dom' is not reset before using it.

We have to be careful with this as that means we MUST have 'dom' set
before pirq_guest_bind() is called. As such we add the
'pirq_dpci->dom = d;' to cover for such cases.

The mechanism to tear it down is more complex as there are two ways it
can be executed:

 a) pci_clean_dpci_irq. This gets called when the guest is being
    destroyed. We end up calling 'tasklet_kill'.

    The scenarios in which the 'struct pirq' (and subsequently the
    'hvm_pirq_dpci') gets destroyed is when:

    - guest did not use the pirq at all after setup.
    - guest did use pirq, but decided to mask and left it in that
      state.
    - guest did use pirq, but crashed.

    In all of those scenarios we end up calling 'tasklet_kill' which
    will spin on the tasklet if it is running.

 b) pt_irq_destroy_bind (guest disables the MSI). We double-check that
    the softirq has run by piggy-backing on the existing
    'pirq_cleanup_check' mechanism which calls 'pt_pirq_cleanup_check'.
    We add the extra call to 'pt_pirq_softirq_active' in
    'pt_pirq_cleanup_check'.

    NOTE: Guests that use event channels unbind first the event channel
    from PIRQs, so the 'pt_pirq_cleanup_check' won't be called as event
    is set to zero. In that case we either clean it up via the a)
    mechanism. It is OK to re-use the tasklet when 'pt_irq_create_bind'
    is called afterwards.

    There is an extra scenario regardless of event being set or not:
    the guest did 'pt_irq_destroy_bind' while an interrupt was
    triggered and tasklet was scheduled (but had not been run). It is
    OK to still run the tasklet as hvm_dirq_assist won't do anything
    (as the flags are set to zero). As such we can exit out of
    hvm_dirq_assist without doing anything.

Suggested-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
dpci: replace tasklet with softirq

The existing tasklet mechanism has a single global spinlock that is
taken every-time the global list is touched. And we use this lock quite
a lot - when we call do_tasklet_work which is called via an softirq and
from the idle loop. We take the lock on any operation on the
tasklet_list.

The problem we are facing is that there are quite a lot of tasklets
scheduled. The most common one that is invoked is the one injecting the
VIRQ_TIMER in the guest. Guests are not insane and don't set the
one-shot or periodic clocks to be in sub 1ms intervals (causing said
tasklet to be scheduled for such small intervalls).

The problem appears when PCI passthrough devices are used over many
sockets and we have an mix of heavy-interrupt guests and idle guests.
The idle guests end up seeing 1/10 of its RUNNING timeslice eaten by
the hypervisor (and 40% steal time).

The mechanism by which we inject PCI interrupts is by hvm_do_IRQ_dpci
which schedules the hvm_dirq_assist tasklet every time an interrupt is
received. The callchain is:

_asm_vmexit_handler
 -> vmx_vmexit_handler
    ->vmx_do_extint
        -> do_IRQ
            -> __do_IRQ_guest
                -> hvm_do_IRQ_dpci
                   tasklet_schedule(&dpci->dirq_tasklet);
                   [takes lock to put the tasklet on]

[later on the schedule_tail is invoked which is 'vmx_do_resume']

vmx_do_resume
 -> vmx_asm_do_vmentry
        -> call vmx_intr_assist
          -> vmx_process_softirqs
            -> do_softirq
              [executes the tasklet function, takes the
               lock again]

While on other CPUs they might be sitting in a idle loop and invoked to
deliver an VIRQ_TIMER, which also ends up taking the lock twice: first
to schedule the v->arch.hvm_vcpu.assert_evtchn_irq_tasklet (accounted
to the guests' BLOCKED_state); then to execute it - which is accounted
for in the guest's RUNTIME_state.

The end result is that on a 8 socket machine with PCI passthrough,
where four sockets are busy with interrupts, and the other sockets have
idle guests - we end up with the idle guests having around 40% steal
time and 1/10 of its timeslice (3ms out of 30 ms) being tied up taking
the lock. The latency of the PCI interrupts delieved to guest is also
hindered.

With this patch the problem disappears completly. That is removing the
lock for the PCI passthrough use-case (the 'hvm_dirq_assist' case) by
not using tasklets at all.

The patch is simple - instead of scheduling an tasklet we schedule our
own softirq - HVM_DPCI_SOFTIRQ, which will take care of running
'hvm_dirq_assist'. The information we need on each CPU is which
'struct hvm_pirq_dpci' structure the 'hvm_dirq_assist' needs to run on.
That is simple solved by threading the 'struct hvm_pirq_dpci' through a
linked list. The rule of only running one 'hvm_dirq_assist' for only
one 'hvm_pirq_dpci' is also preserved by having 'schedule_dpci_for'
ignore any subsequent calls for an domain which has already been
scheduled.

== Code details ==

Most of the code complexity comes from the '->dom' field in the
'hvm_pirq_dpci' structure. We use it for ref-counting and as such it
MUST be valid as long as STATE_SCHED bit is set. Whoever clears the
STATE_SCHED bit does the ref-counting and can also reset the '->dom'
field.

To compound the complexity, there are multiple points where the
'hvm_pirq_dpci' structure is reset or re-used. Initially (first time
the domain uses the pirq), the 'hvm_pirq_dpci->dom' field is set to
NULL as it is allocated. On subsequent calls in to 'pt_irq_create_bind'
the ->dom is whatever it had last time.

As this is the initial call (which QEMU ends up calling when the guest
writes an vector value in the MSI field) we MUST set the '->dom' to a
the proper structure (otherwise we cannot do proper ref-counting).

The mechanism to tear it down is more complex as there are three ways
it can be executed. To make it simpler everything revolves around
'pt_pirq_softirq_active'. If it returns -EAGAIN that means there is an
outstanding softirq that needs to finish running before we can continue
tearing down. With that in mind:

a) pci_clean_dpci_irq. This gets called when the guest is being
   destroyed. We end up calling 'pt_pirq_softirq_active' to see if it
   is OK to continue the destruction.

   The scenarios in which the 'struct pirq' (and subsequently the
   'hvm_pirq_dpci') gets destroyed is when:

   - guest did not use the pirq at all after setup.
   - guest did use pirq, but decided to mask and left it in that state.
   - guest did use pirq, but crashed.

   In all of those scenarios we end up calling 'pt_pirq_softirq_active'
   to check if the softirq is still active. Read below on the
   'pt_pirq_softirq_active' loop.

b) pt_irq_destroy_bind (guest disables the MSI). We double-check that
   the softirq has run by piggy-backing on the existing
   'pirq_cleanup_check' mechanism which calls 'pt_pirq_cleanup_check'.
   We add the extra call to 'pt_pirq_softirq_active' in
   'pt_pirq_cleanup_check'.

   NOTE: Guests that use event channels unbind first the event channel
   from PIRQs, so the 'pt_pirq_cleanup_check' won't be called as 'event'
   is set to zero. In that case we either clean it up via the a) or c)
   mechanism.

   There is an extra scenario regardless of 'event' being set or not:
   the guest did 'pt_irq_destroy_bind' while an interrupt was triggered
   and softirq was scheduled (but had not been run). It is OK to still
   run the softirq as hvm_dirq_assist won't do anything (as the flags
   are set to zero). However we will try to deschedule the softirq if
   we can (by clearing the STATE_SCHED bit and us doing the
   ref-counting).

c) pt_irq_create_bind (not a typo). The scenarios are:

   - guest disables the MSI and then enables it (rmmod and modprobe in
     a loop). We call 'pt_pirq_reset' which checks to see if the
     softirq has been scheduled. Imagine the 'b)' with interrupts in
     flight and c) getting called in a loop.

We will spin up on 'pt_pirq_is_active' (at the start of the
'pt_irq_create_bind') with the event_lock spinlock dropped and waiting
(cpu_relax). We cannot call 'process_pending_softirqs' as it might
result in a dead-lock. hvm_dirq_assist will be executed and then the
softirq will clear 'state' which signals that that we can re-use the
'hvm_pirq_dpci' structure. In case this softirq is scheduled on a
remote CPU the softirq will run on it as the semantics behind an
softirq is that it will execute within the guest interruption.

   - we hit once the error paths in 'pt_irq_create_bind' while an
     interrupt was triggered and softirq was scheduled.

If the softirq is in STATE_RUN that means it is executing and we should
let it continue on. We can clear the '->dom' field as the softirq has
stashed it beforehand. If the softirq is STATE_SCHED and we are
successful in clearing it, we do the ref-counting and clear the '->dom'
field. Otherwise we let the softirq continue on and the '->dom' field
is left intact. The clearing of the '->dom' is left to a), b) or again
c) case.

Note that in both cases the 'flags' variable is cleared so
hvm_dirq_assist won't actually do anything.

Suggested-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
adjust number of domains in cpupools when destroying domain

Commit bac6334b51d9bcfe57ecf4a4cb5288348fcf044a (move domain to
cpupool0 before destroying it) introduced an error in the accounting
of cpupools regarding the number of domains. The number of domains
is nor adjusted when a domain is moved to cpupool0 in kill_domain().

Correct this by introducing a cpupool function doing the move
instead of open coding it by calling sched_move_domain().

Reported-by: Dietmar Hahn <dietmar.hahn@ts.fujitsu.com>
Signed-off-by: Juergen Gross <jgross@suse.com>
Tested-by: Dietmar Hahn <dietmar.hahn@ts.fujitsu.com>
Reviewed-by: Andrew Cooper <Andrew.Cooper3@citrix.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
tools: various things just for fast test build

- Remove python prefix
- Add/remove/stop of main services (xencommons, xendomains)

NOTE:
This patch is only for do fast test build with make debball
and upstream qemu build with spice full features.

Signed-off-by: Fabio Fantoni <fabio.fantoni@m2r.biz>

---

Changes from v2:
- remove add of spice and usbredir on upstream qemu build

Changes from v1:
- Added: Add/remove/stop of main services
libxl: Add qxl vga interface support for upstream qemu

Usage:
vga="qxl"

Qxl vga support many resolutions that not supported by stdvga,
mainly the 16:9 ones and other high up to 2560x1600.
With QXL you can get improved performance and smooth video also
with high resolutions and high quality.
Require their drivers installed in the domU and spice used
otherwise act as a simple stdvga.

Signed-off-by: Fabio Fantoni <fabio.fantoni@m2r.biz>
Signed-off-by: Zhou Peng <zpengxen@gmail.com>
Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>

---

Changes in v17:
- refresh

Changes in v16:
- refresh
- improved commit description

Changes in v15:
- refresh
- small code improvements in libxl_dm.c

Changes in v14:
- refresh
- update qemu parameters (from -vga to -device)

NOTES:
Works correctly with windows domUs, tested on windows 7 64 bit
with qxl driver from spice guest tools 0.74.
I tested some resolution not supported by stdvga (1366x768, 1600x900
and 1920x1080) with 32 bit color and all works good equal to kvm.
For now not works on linux domUs when xorg have 100% cpu and black
screen with qxl driver installed.
Seems needed other changes/fixes on xen and/or xorg/qxl driver side
before have it full working with linux domUs.
libxl: add basic spice support for pv domUs

This patch adds basic spice support for pv domUs.
The qemu parameters are the same as the hvm ones and they works.
Therefore xl cfg parameters are the same as the hvm ones except that
features not supported yet by pv domUs (vdagent and usbredirection)
are kept disabled by default.
It also enables vfb and vkb required to have basic spice working.

Signed-off-by: Fabio Fantoni <fabio.fantoni@m2r.biz>

---

Notes:
- The vfb part is only a draft and needs to be improved.
- Patch is tested and working, except for the pointer not
visible in some cases with pv domUs but always working.
- I not use the api, test the u.hvm.spice retro-compatibility with
api is needed.

Any feedback is appreciated.

Changes in v6:
- refresh libxl_create.c

Changes in v5:
- libxl_create.c: * don't copy u.hvm.spice in the newer if
the newer is already used
* set default for all spice bool options in any case
* spice features not supported in pv will be disabled and
will show a warning about them if was setted enabled
- xl_cmdimpl.c: parse all spice options out of hvm part
- libxl_dm.c: changed some forgotten u.hvm.spice to spice

Changes in v4:
- added libxl.h changes
- libxl_create.c: added older u.hvm.spice compatibility
copying it in newer one

Changes in v3:
- xl.cfg.pod.5: moved spice out of hvm section and specified
the features for now hvm only.
- libxl_types.idl: added spice struct out of keyedunion hvm only.
- use new generic spice struct instead of hvm only ones.

Changes in v2:
- xl_cmdimpl.c: always set vnc and sdl toplevel parameters in &vfb
with vnc or spice enabled on pv domUs otherwise in some cases it
would fail with error for one bool default value missing.
- libxl_dm.c: do not add -nographic if spice is enabled, even though
-nographic seems buggy in upstream qemu.
libxl: Spice image compression setting support for upstream qemu

Usage:
spice_image_compression=[auto_glz|auto_lz|quic|glz|lz|off]

Specifies what image compression is to be used by spice (if given),
otherwise the qemu default will be used.

Signed-off-by: Fabio Fantoni <fabio.fantoni@m2r.biz>
libxl: Spice streaming video setting support for upstream qemu

Usage:
spice_streaming_video=[filter|all|off]

Specifies what streaming video setting is to be used by spice (if given),
otherwise the qemu default will be used.

Signed-off-by: Fabio Fantoni <fabio.fantoni@m2r.biz>
Use updated ovmf upstream git

NOTE:
This is for test only, even if update ovmf introduce xen pv block
support and many ovmf fixes may introduce also regressions.

Signed-off-by: Fabio Fantoni <fabio.fantoni@m2r.biz>
x86/hvm: Extend HVM cpuid leaf with vcpu id

To perform certain hypercalls HVM guests need to use Xen's idea of
vcpu id, which may well not match the guest OS idea of CPU id.
This patch adds vcpu id to the HVM cpuid leaf allowing the guest
to build a mapping.

Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
Cc: Keir Fraser <keir@xen.org>
Cc: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
x86/hvm: Add per-vcpu evtchn upcalls

HVM guests have always been confined to using the domain callback
via (see HVM_PARAM_CALLBACK_IRQ) to receive event notifications.
This is usually an IOAPIC vector and is only used if the event
channel is bound to vcpu 0.

PV-on-HVM Linux uses a local APIC vector for event channel upcall,
set using HVM_PARAM_CALLBACK_IRQ by ORing in a special bit (bit 57)
into the value (see params.h), but this mechanism is not suitable
in the general case since the vector must be the same on all vcpus
which cannot be guaranteed on some OS (e.g. Windows).

This patch adds a new HVM op allowing a guest to specify a local
APIC vector to use as an upcall notification for a specific vcpu
therefore coping with the case of differing vector numbers.

Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
Cc: Keir Fraser <keir@xen.org>
Cc: Jan Beulich <jbeulich@suse.com>