iommu/crash: Interrupt remapping is also disabled on crash

This fixes a regression side-effect caused by:
  IOMMU: properly check whether interrupt remapping is enabled
    git: fae0372140befb88d890a30704a8ec058c902af8
     hg: 26742:e1ec14bad0cb

On the crash path in nmi_shootdown_cpus(), we shut down the IOMMU, then
disable the IOAPIC.

On systems which support interrupt remapping, the variable iommu_intremap
remains set, meaning that disable_IO_APIC() issues interrupt remapping
invalidate requests.

IOAPIC interrupt remapping used to be conditional on iommu_enabled, but is now
conditional on iommu_intremap, following the above changeset.

This behaviour can be fixed by also indicating that interrupt remapping is not
enabled after shutting down the IOMMU.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
VTD: Remove the check for reserved device scope type

Though we only have four valid types now, the new type may be added in future.
It's better to remove the check and only deal with the type that we can
recognize.

Signed-off-by: Yang Zhang <yang.z.zhang@Intel.com>
Signed-off-by: Xiantao Zhang <xiantao.zhang@Intel.com>
Acked-by: Keir Fraser <keir@xen.org>

Add log message for this case.

Signed-off-by: Jan Beulich <jbeulich@suse.com>
xc: use XENMEM_claim_pages hypercall during guest creation.

We add an extra parameter to the structures passed to the
PV routine (arch_setup_meminit) and HVM routine (setup_guest)
that determines whether the claim hypercall is to be done.

The contents of the 'claim_enabled' is defined as an 'int'
in case the hypercall expands in the future with extra
flags (for example for per-NUMA allocation). For right now
the proper values are: 0 to disable it or 1 to enable
it.

If the hypervisor does not support this function, the
xc_domain_claim_pages and xc_domain_get_outstanding_pages
will silently return 0 (and set errno to zero).

Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
[v2: Updated per Ian's recommendations]
[v3: Added support for out-of-sync hypervisor]
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
xl: Implement XENMEM_claim_pages support via 'claim_mode' global config

The XENMEM_claim_pages hypercall operates per domain and it should be
used system wide. As such this patch introduces a global configuration
option 'claim_mode' that by default is disabled.

If this option is enabled then when a guest is created there will be an
guarantee that there is memory available for the guest. This is an
particularly acute problem on hosts with memory over-provisioned guests
that use tmem and have self-balloon enabled (which is the default option
for them). The self-balloon mechanism can deflate/inflate the balloon
quickly and the amount of free memory (which 'xl info' can show) is stale
the moment it is printed. When claim is enabled a reservation for the
amount of memory ('memory' in guest config) is set, which is then reduced
as the domain's memory is populated and eventually reaches zero.

If the reservation cannot be meet the guest creation fails immediately
instead of taking seconds/minutes (depending on the size of the guest)
while the guest is populated.

Note that to enable tmem type guests, one needs to provide 'tmem' on the
Xen hypervisor argument and as well on the Linux kernel command line.

There are two boolean options:

(0) No claim is made. Memory population during guest creation will be
attempted as normal and may fail due to memory exhaustion.

(1) Normal memory and freeable pool of ephemeral pages (tmem) is used when
calculating whether there is enough memory free to launch a guest.
This guarantees immediate feedback whether the guest can be launched due
to memory exhaustion (which can take a long time to find out if launching
massively huge guests) and in parallel.

[v1: Removed own claim_mode type, using just bool, improved docs, all per
Ian's suggestion]
[v2: Updated the comments]
[v3: Rebase on top 733b9c524dbc2bec318bfc3588ed1652455d30ec (xl: add vif.default.script)]
[v4: Fixed up comments]
[v5: s/global_claim_mode/claim_mode/]
[v6: Ian Jackson's feedback: use libxl_defbool, better comments, etc]
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
xl: 'xl info' print outstanding claims if enabled (claim_mode=1 in xl.conf)

This patch provides the value of the currently outstanding pages
claimed for all domains. This is a total global value that influences
the hypervisors' MM system.

When a claim call is done, a reservation for a specific amount of pages
is set and also a global value is incremented. This global value is then
reduced as the domain's memory is populated and eventually reaches zero.
The toolstack (libxc) also sets the domain's claim to zero when the population
of memory has completed as an extra step. Any call to destroy the domain
will also set the domain's claim to zero.

If the reservation cannot be meet the guest creation fails immediately
instead of taking seconds or minutes (depending on the size of the guest)
while the toolstack populates memory.

See patch: "xl: Implement XENMEM_claim_pages support via 'claim_mode'
global config" for details on how it is implemented.

The value fluctuates quite often so the value is stale once it is provided
to the user-space.  However it is useful for diagnostic purposes.

It is only printed when the global "claim_mode" option in xl.conf(5)
is set to enabled (1). The 'man xl' shows the details of this item.

[v1: s/unclaimed/outstanding/]
[v2: Made libxl_get_claiminfo return just MemKB suggested by Ian Campbell]
[v3: Made libxl_get_claininfo return MemMB to conform to the other values printed]
[v4: Improvements suggested by Ian Jackson, also added docs to xl.pod.1]
[v5: Clarify how claims are cancelled, split >72 characters - Ian Jackson]
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
xc: export outstanding_pages value in xc_dominfo structure.

This patch provides the value of the currently outstanding pages
claimed for a specific domain. This is a value that influences
the global outstanding claims value (See patch: "xl: 'xl info'
print outstanding claims if enabled") returned via
xc_domain_get_outstanding_pages hypercall. This domain value
decrements as the memory is populated for the guest and
eventually reaches zero.

This patch is neccessary for "xl: export 'outstanding_pages' value
from xcinfo" patch.

Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
[v2: s/unclaimed_pages/outstanding_pages/ per Tim's suggestion]
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
xl: export 'outstanding_pages' value from xcinfo

This patch provides the value of the currently outstanding pages
claimed for a specific domain. This is a value that influences
the global outstanding claims value (See patch: "xl: 'xl info'
print outstanding claims if enabled") returned via
xc_domain_get_outstanding_pages hypercall. This domain value
decrements as the memory is populated for the guest and
eventually reaches zero.

With this patch it is possible to utilize this field.

Acked-by: Ian Campbell <ian.campbell@citrix.com>
[v2: s/unclaimed/outstanding/ per Tim's suggestion]
[v3: Don't use SXP printout file per Ian's suggestion]
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
xl: 'xl claims' print outstanding per domain claims

This is similar to "xl: 'xl info' print outstanding claims if enabled
(claim_mode=1 in xl.conf)" which exposes the global claim value.

This patch provides the value of the currently outstanding pages
claimed for each domains. This is per domain value which is added
to the global claim value which influences the hypervisors' MM system.

When a claim call is done, a reservation for a specific amount of pages
is set (and this patch lists said number) and also a global value is
incremented. This global value is then reduced as the domain's memory
is populated and eventually reaches zero.

The toolstack (libxc) also sets the domain's claim to zero when the population
of memory has completed as an extra step. Any call to destroy the domain
will also set the domain's claim to zero.

If the reservation cannot be meet the guest creation fails immediately
instead of taking seconds or minutes (depending on the size of the guest)
while the toolstack populates memory.

See patch: "xl: Implement XENMEM_claim_pages support via 'claim_mode'
global config" for details on how it is implemented.

The value fluctuates quite often so the value is stale once it is provided
to the user-space.  However it is useful for diagnostic purposes.

It is printed irregardless of global "claim_mode" option in xl.conf(5).
That is b/c the user might have enabled, launched a guest, and then
disabled the option - and we should still report the correct outstanding
claim value.  The 'man xl' shows the details of this argument.
The output is close to what 'xl list' looks like:

Name                                        ID   Mem VCPUs      State   Time(s)  Claimed
Domain-0                                     0  2047     4     r-----      19.7     0
OL5                                          2  2048     1     --p---       0.0   847
OL6                                          3  1024     4     r-----       5.9     0
Windows_XP                                   4  2047     1     --p---       0.0  1989

[In which it can be seen that the OL5 guest still has 847MB of claimed
memory (out of the total 2048MB where 1191MB has been allocated to
the guest).]

Please note that the 'Mem' column has the cumulative value of outstanding
claims and the total amount of memory that has been allocated to the guest.

[v1: claims, not claim-list]
[v2: Add outstanding and current memkb in the output list]
[v3: Clairy docs and relax some checks]
[v4: Removed comments about guest config memory being the same as 'Mem']
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
xl: Fix 'free_memory' to include outstanding_claims value.

Updating to make it clear that free_memory reported by 'xl info'
is influenced by the outstanding claim value. That is the free
memory that will be available to the host once all outstanding
claims have been completed. This modifies the behavior that the
patch titled "xl: 'xl info' print outstanding claims if enabled
(claim_mode=1 in xl.conf)" had - which reported the
outstanding claims and nothing else.

The free_pages as reported by the hypervisor is the currently
available count of pages on the heap. The outstanding pages is
the total amount of pages reserved for guests (so not taken from
the heap yet). As guests are being populated the memory from the
heap shrinks and the outstanding count of pages decreases.
The total memory used for guests increases.

As the available count of pages on the heap and outstanding
claims are intertwined, report the amount of free memory available
to be a combination of that. That is free heap memory minus the
outstanding pages.

We also make some odd choices in reporting. By default we will
only display 'outstanding_claims' if the claim_mode is enabled
in the global configuration file. However, if there are outstanding
claims, we will ignore the claim_mode and report these values.

Suggested-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
x86: allow VCPUOP_register_vcpu_info to work again on PVHVM guests

For details on the hypercall please see commit
c58ae69360ccf2495a19bf4ca107e21cf873c75b (VCPUOP_register_vcpu_info) and
the c/s 23143 (git commit 6b063a4a6f44245a727aa04ef76408b2e00af9c7)
(x86: move pv-only members of struct vcpu to struct pv_vcpu)
that introduced the regression.

The current code allows the PVHVM guest to make this hypercall.
But for PVHVM guest it always returns -EINVAL (-22) for Xen 4.2
and above. Xen 4.1 and earlier worked.

The reason is that the check in map_vcpu_info would fail
at:

  if ( v->arch.vcpu_info_mfn != INVALID_MFN )

The reason is that the vcpu_info_mfn for PVHVM guests ends up by
defualt with the value of zero (introduced by c/s 23143).

The code in vcpu_initialise which initialized vcpu_info_mfn to a
valid value (INVALID_MFN), would never be called for PVHVM:

    if ( is_hvm_domain(d) )
    {
        rc = hvm_vcpu_initialise(v);
        goto done;
    }

    v->arch.pv_vcpu.vcpu_info_mfn = INVALID_MFN;

while previously it would be:

     v->arch.vcpu_info_mfn = INVALID_MFN;

[right at the start of the function in Xen 4.1]

This fixes the problem with Linux advertising this error:
register_vcpu_info failed: err=-22

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
xen, libxc: rename xenctl_cpumap to xenctl_bitmap

More specifically:
 1. replaces xenctl_cpumap with xenctl_bitmap
 2. provides bitmap_to_xenctl_bitmap and the reverse;
 3. re-implement cpumask_to_xenctl_bitmap with
    bitmap_to_xenctl_bitmap and the reverse;

Other than #3, no functional changes. Interface only slightly
afected.

This is in preparation of introducing NUMA node-affinity maps.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Acked-by: Juergen Gross <juergen.gross@ts.fujitsu.com>
Acked-by: Keir Fraser <keir@xen.org>
xen, libxc: introduce xc_nodemap_t

And its handling functions, following suit from xc_cpumap_t.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Acked-by: Juergen Gross <juergen.gross@ts.fujitsu.com>
Acked-by: Keir Fraser <keir@xen.org>
xen: sched_credit: when picking, make sure we get an idle one, if any

The pcpu picking algorithm treats two threads of a SMT core the same.
More specifically, if one is idle and the other one is busy, they both
will be assigned a weight of 1. Therefore, when picking begins, if the
first target pcpu is the busy thread (and if there are no other idle
pcpu than its sibling), that will never change.

This change fixes this by ensuring that, before entering the core of
the picking algorithm, the target pcpu is an idle one (if there is an
idle pcpu at all, of course).

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: Juergen Gross <juergen.gross@ts.fujitsu.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Acked-by: Keir Fraser <keir@xen.org>
xen: sched_credit: let the scheduler know about node-affinity

As vcpu-affinity tells where VCPUs must run, node-affinity tells
where they prefer to. While respecting vcpu-affinity remains mandatory,
node-affinity is not that strict, it only expresses a preference,
although honouring it will bring significant performance benefits
(especially as compared to not having any affinity at all).

This change modifies the VCPUs load balancing algorithm (for the
credit scheduler only), introducing a two steps logic. During the
first step, we use both the vcpu-affinity and the node-affinity
masks (by looking at their intersection). The aim is giving precedence
to the PCPUs where the domain prefers to run, as expressed by its
node-affinity (with the intersection with the vcpu-afinity being
necessary in order to avoid running a VCPU where it never should).
If that fails in finding a valid PCPU, the node-affinity is just
ignored and, in the second step, we fall back to using cpu-affinity
only.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: Juergen Gross <juergen.gross@ts.fujitsu.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Acked-by: Keir Fraser <keir@xen.org>
xen: allow for explicitly specifying node-affinity

Make it possible to pass the node-affinity of a domain to the hypervisor
from the upper layers, instead of always being computed automatically.

Note that this also required generalizing the Flask hooks for setting
and getting the affinity, so that they now deal with both vcpu and
node affinity.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Acked-by: Juergen Gross <juergen.gross@ts.fujitsu.com>
Acked-by: Keir Fraser <keir@xen.org>
libxc: allow for explicitly specifying node-affinity

By providing the proper get/set interface and wiring them
to the new domctl-s from the previous commit.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: Juergen Gross <juergen.gross@ts.fujitsu.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
libxl: allow for explicitly specifying node-affinity

By introducing a nodemap in libxl_domain_build_info and
providing the get/set methods to deal with it.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: Juergen Gross <juergen.gross@ts.fujitsu.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
libxl: optimize the calculation of how many VCPUs can run on a candidate

For choosing the best NUMA placement candidate, we need to figure out
how many VCPUs are runnable on each of them. That requires going through
all the VCPUs of all the domains and check their affinities.

With this change, instead of doing the above for each candidate, we
do it once for all, populating an array while counting. This way, when
we later are evaluating candidates, all we need is summing up the right
elements of the array itself.

This reduces the complexity of the overall algorithm, as it moves a
potentially expensive operation (for_each_vcpu_of_each_domain {})
outside from the core placement loop, so that it is performed only
once instead of (potentially) tens or hundreds of times. More
specifically, we go from a worst case computation time complaxity of:

 O(2^n_nodes) * O(n_domains*n_domain_vcpus)

To, with this change:

 O(n_domains*n_domains_vcpus) + O(2^n_nodes) = O(2^n_nodes)

(with n_nodes<=16, otherwise the algorithm suggests partitioning with
cpupools and does not even start.)

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: Juergen Gross <juergen.gross@ts.fujitsu.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
libxl: automatic placement deals with node-affinity

Which basically means the following two things:
 1) during domain creation, it is the node-affinity of
    the domain --rather than the vcpu-affinities of its
    VCPUs-- that is affected by automatic placement;
 2) during automatic placement, when counting how many
    VCPUs are already "bound" to a placement candidate
    (as part of the process of choosing the best
    candidate), both vcpu-affinity and node-affinity
    are considered.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Acked-by: Juergen Gross <juergen.gross@ts.fujitsu.com>
xl: add node-affinity to the output of `xl list`

Node-affinity is now something that is under (some) control of the
user, so show it upon request as part of the output of `xl list'
by the `-n' option.

Re the patch, the print_bitmap() related hunk is _mostly_ code motion,
although there is a very minor change in the code, basically to allow
using the function for printing both cpu and node bitmaps (as, in case
all bits are sets, it used to print "any cpu", which doesn't fit the
nodemap case).

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: Juergen Gross <juergen.gross@ts.fujitsu.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>