HTools: Pass --ignore-soft-errors all the way through

Fixes hail silently ignoring the --ignore-soft-errors flag.

See issue 1006.

For hspace and tests, the default is to use defaultOptions
(do not force, like it was before).

The core change is in `allocateOnSingle`/`allocateOnPair`,
which now call `addPriEx force` instead of `addPri` (that one
defaults to `force = False`).

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Fix internal consistency in test data

The example cluster was inconsistent: the amount of memory
claimed to be used by the instances was non-zero (in fact,
huge) even though there were no instances on the node.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Add generator for empty nodes

For some tests, it is not only relevant that a node is online
but also that it is internally in a consistent state. Add a
generator for this.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Only test serialization on consistent clusters

In the text backend, some information about the node,
like the actual free memory, is implicit and only
computed from other data. Therefore, for deserialization
to obtain the same cluster that we had originally, insist
that the original data be internally consistent.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Signed-off-by: Niklas Hambuechen <niklash@google.com>
htools: make checkData idempotent

The usual understanding of a check function is that
it does not modify the underlying data as to achieve
an inconsistent state. However, by the unfortunate
combination of checking (and fixing xMem) and "reserving"
memory for the down instances by reducing the amount
of free memory on the node that property was violated.

Make checkData again a pure check (that only fixes
values and hence behaves in an idempotent way). Memory
needs for the down instances will be taken into account
together with the memory usage of the forthcoming instances.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Signed-off-by: Niklas Hambuechen <niklash@google.com>
In tests deserialise as we do in practise

i.e., not only merging the data but also checking it.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Signed-off-by: Niklas Hambuechen <niklash@google.com>
Loader: Refactor node updating

Infix style makes it easier to add more node modifications.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Tests: Fix some flipped ==?

The expected value should be on the right.

Also fixed usage of `il'` when `il` was a free name.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
HTools.Backend.Text: Allow "forthcoming" instances

The text format is extended by a column; "Y" stands for "the instance
is forthcoming" and "N" stands for "the instance is real".

Instances in older versions of the data format are treated as real
since so far instances could not be forthcoming.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
HTools: Node violation checks for forthcoming instances

This implements basic support for forthcoming instances in htools,
in particular whether a forthcoming instance to be placed fits on it
and the maintainance of the relevant statistics to decide this.

Each statistic in the `Node` data type that has to handle forthcoming
instances gets a `somethingForth` equivalent that includes the
corresponding `something` statistic for real instances.

For example, when `fMem` measures the free memory taking all real
instances into account, `fMemForth` is the free memory on the node
in case all forthcoming instances on the node were also made real,
thus being less or equal than `fMem` at any given time.

This commit does NOT yet
- handle the maintenance of rMemForth, failN1Forth, and peersForth,
  so n+1 safety for forthcoming instances is not yet implemented
- add the new `somethingForth` fields to the cluster balancedness
  measure (`detailedCVInfoExt`)
- add `test/data/htools/*.json` or `*.data` files that test this
  feature

All unit tests pass.

Signed-off-by: Niklas Hambuechen <niklash@google.com>