Add design doc for performance tests

This design doc describes which tests are added in order to test the
performance of Ganeti, specifically when handling multiple jobs in
parallel.

Note that this design doc is submitted to stable-2.10 so performance
changes over different Ganeti versions can be captured.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Add a utility to combine shell commands

Add a function that combines individual shell commands into
a single command (calling the standard shell) that executes
the given commands in sequence while they succeed.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Set correct Ganeti version on setup commands

When asked to execute a setup command, prefix it with
a command sequence to test for the existence of the
needed Ganeti version and switching to it.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Add design-node-security.rst to docinput

Currently, design-node-security.rst is not included in the dist tarball,
making it impossible to rebuild the documentation using only the tarball.

Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Merge branch 'stable-2.10' into stable-2.11

* stable-2.10
  Set correct Ganeti version on setup commands
  Add a utility to combine shell commands
  Add design doc for performance tests

Conflicts:
	Makefile.am
	doc/design-draft.rst
	lib/bootstrap.py
Resolution:
	always take all additions

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Add a test for parsing version strings

...even in the presence of patch levels.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
kvm: use a dedicated QMP socket for kvmd

The KVM daemon keeps a persistent connection to the instances' QMP
sockets, listening for asynchronous events. As each monitor socket
(either human, or QMP) can handle only one client at a time, this has
the side-effect that QMP cannot be used for regular instance operations.

We fix this by adding a dedicated QMP socket for use by kvmd and
changing kvmd to look only for sockets with the specific extension.

Note that this change means that kvmd will not be able to monitor
already running instances spawned by older versions of Ganeti.

Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>
Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Add hsqueeze to NEWS

As the release of the 2.11 series added a new program,
it should be mentioned.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Recursively clear serial numbers

Disk objects, in general, are of recursive nature. Therefore,
when downgrading them, do so recursively.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Add configure option to pass GHC flags

Adding the HEXTRA option to make might not be practical for a change
that should be always applied, e.g., hiding a certain package. This
patch allows the flags to be specified at the configure level.

Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
KVM: set IFF_ONE_QUEUE on created tap interfaces

The IFF_ONE_QUEUE flag directs the kernel to only queue tap packets once
(as opposed to queueing them twice, once for the device, and once for the
qdisc), possibly avoiding interface stalls when one of the queues overruns.

This is the default behaviour of the kernel tun/tap driver in Linux
>= 3.8. Also, qemu >= 1.5.0 sets this flag when opening the tap device
itself (but not for tap interfaces inherited via fds), according to this
commit:

  commit d26e445c80fddcc7483b83f3115e5067fef28fe6
  Author: Peter Lieven <pl@dlhnet.de>
  Date:   Mon Feb 25 10:17:08 2013 +0100

      tap: set IFF_ONE_QUEUE per default

      historically the kernel queues packets two times. once
      at the device and second in qdisc. this is believed to cause
      interface stalls if one of these queues overruns.

      setting IFF_ONE_QUEUE is the default in kernels >= 3.8. the
      flag is ignored since then. see kernel commit
      5d097109257c03a71845729f8db6b5770c4bbedc

      Signed-off-by: Peter Lieven <pl@kamp.de>
      Acked-by: Michael S. Tsirkin <mst@redhat.com>
      Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

Since we handle tap creation ourselves, we duplicate this behavior in
_OpenTap. IFF_ONE_QUEUE has been there as far back as the kernel's git
history goes (2.6.12), so it is safe to add the flag unconditionally.
Also the flag is invisible to the guest and will not break migrations of
already running instances.

Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Add an example for node evacuation

The configuration shows an unbalanced cluster with
a node being drained. The natural evacuation strategy
includes frf-moves.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Add an option for restricted migration

This option will allow node evacuation with migrations only
off the nodes to be evacuated.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Support restricted migration

Make hbal support an option to disallow ReplacePrimary moves
and restrict ReplaceAndFailover to instances where the primary
node is drained. If used in evacuation mode, the only migration
moves will be off the drained nodes.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Describe the --restricted-migration option

Add the --restricted-migration option to the man page together with
a hint on the intended use case.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Add a basic test for --restricted-migration

Essentially verify that, in the given example, a solution is still
found and that the original present failover is dropped.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Merge branch 'stable-2.9' into stable-2.10

* stable-2.9
  Add configure option to pass GHC flags
  KVM: set IFF_ONE_QUEUE on created tap interfaces

Conflicts:
	configure.ac # Taken both contributions

Signed-off-by: Hrvoje Ribicic <riba@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Fail in replace-disks if attaching disks fails

Previously, if attaching the new secondary during a replace-disks
operations failed, only a warning was emitted. The subsequent sync-disks
operation cannot finish in such a case, however.

Therefore, this patch changes the warning into an error. This way it's
possible for administrators to fix the problem and the cluster does not
remain in a blocked state.

This fixes issue 814.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Test parallel instance creation and removal

This is the first performance related test. It creates as many instances
as available in the QA config in parallel and removes them (again in
parallel) immediately after the creation succeeded.

In order to ease writing of additional tests, a lot of the logic is kept
in helper classes.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Test parallel node-count instance creation

Test the parallel creation (and removal) of as many instances as there
are nodes in the cluster.

Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>