Split cluster.py

cluster.py is nearly 4000 lines long and consists of two independent
parts: on one hand there is everything related to verifying the cluster
and there is everything else.

This change moves everything from cmdlib/cluster.py to either
cmdlib/cluster/__init__.py or cmdlib/cluster/verify.py.

Signed-off-by: Aaron Karper <akarper@google.com>
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Reduce maximal module size

This patch reduces the max-module-size of the project to 3600, which is
slightly larger than the currently largest module (baring backend.py,
which got a free pass): config.py

In an efford to reduce the module size to something more reasonable, the
next person to hit the wall will find a nice reminder: Please reduce the
maximal module size further when possible.

Signed-off-by: Aaron Karper <akarper@google.com>
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Tests: Fix bad bad alignment style

Fixes an accident when formatting a record.

Signed-off-by: Niklas Hambuechen <niklash@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Add prefix for WConfD livelock file name

WConfD will also need a livelock to request locks in its
own name, e.g., when providing short-lived locks.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Aaron Karper <akarper@google.com>
Move unpacking of config-change to a separate function

In this way, the logic of determining whether the configuration needs
to be distributed, as well as the implicit serial numb can be shared
with other functions.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Aaron Karper <akarper@google.com>
Support config-changes with temporary ConfigLock

Support changes of the configuration with a temporarily acquired
configuration lock. This can be used to support requests to directly
change parts of the configuration.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Aaron Karper <akarper@google.com>
Extend modifyConfig to also modify TempResState

Extend the modifyConfig function to also support modifications
of the state of temporary reservations.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Aaron Karper <akarper@google.com>
Extend config modification to depend on TempRes

Extend modifyConfigWithLock so that the config modification
may also depend on the state of the temporary reservations.
This is needed, e.g., to commit temporary IP reservations.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Aaron Karper <akarper@google.com>
Make WConfD export a function to add an instance

This function handles all the configuration modifications
necessary: it adds the instance to the list of instances,
drops the DRBD minor reservations, and commits the temporary
IPs.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Aaron Karper <akarper@google.com>
Add instances to the configuration directly via WConfD

Make jobs adding instances do so by means of the corresponding
WConfD function. In this way, no long-lived config lock has to
be acquired.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Aaron Karper <akarper@google.com>
Ingore "Use import/export shortcut"

We occasionally reexport modules to keep interfaces stable, most prominently
in Ganeti.Objects. To keep the export statement readable it is helpful to
explicitly name all the modules reexported. This lint warning would make this
impossible.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Fix typo in error in bootstrap

Signed-off-by: Aaron Karper <akarper@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Add cabal file generation to Makefile.am

This allows us to specify the exact version ranges of dependencies we
support, across all operating systems.

See issue #988.

This commit only adds the cabal file and declares the dependencies.
Enforcing them will be subject of a later commit.

So far, we have only given the names of dependency in most cases,
making sure that Ganeti builds with the dependency versions that are in
Debian.
This has caused problems for users that were not running Debian, or
when there were packages installed in the compiling user's ~/.ghc
directory, leading to compile-time type errors instead of pre-build
configuration errors; for an example, see:
  https://code.google.com/p/ganeti/issues/detail?id=979#c10

By explicitly listing our dependencies, we make clear which versions
are supported.

Modules and executables listed in the cabal file are automatically
generated from the Makefile; executables are symlinked in an apps/
directory to avoid repeated compilation as explained in:

  http://stackoverflow.com/a/6711739/263061
  http://stackoverflow.com/q/12305970/263061

For example, in apps/ there is:

  hluxid.hs -> ../src/hluxid.hs
  hluxid.hs.stamp
  [...]

Each executable symlink has a stamp file to track its modification time
because make follow symlinks and thus does not see symlink modification
times.

In addition to declaring our dependencies, this setup allows building
Ganeti's Haskell code with cabal to support editor integration and
cabal-based tooling, the Makefile remains the default way to build Ganeti.
Cabal: Allow json-0.8

Compilation was tested successfully.
Cabal: Allow attoparsec <= 0.12 and lens <= 4.3

Compilation was tested successfully.
Cabal: Allow network <= 2.6

Compilation was tested successfully with network-2.5.0.0 and network-2.6.0.2.
Cabal: Relax lower bounds

Compilation was tested successfully on our buildbot.

Indirect dependencies were removed, found by the `packdeps` tool.
Use Cabal to enforce dependency versions

This uses `cabal configure` to determine which exact dependency versions
we are compiling against, and ensures that these versions are used
by passing -package-id flags to GHC.

The `cabal configure` step makes the build fail before compiling / type
checking if the user tries to compile against a dependency version we don't
support; before, this case led to type errors which were not clearly
user errors. This fixes issue #988.

The output of `cabal configure` is also used to generate MIN_VERSION_*
macros.

MIN_VERSION_* macros are the standard way to build CPP dependency switches
in Haskell packages, and they replace our custom macros (like PARALLEL3
and NO_REGEX_PCRE) which had to be hand-built for each dependency.
We can now query the version of any Haskell dependency without having
to manually add a flag via autoconf.

All ghc and hlint invocations were adjusted to take these macros into
account.

This change introduces a Haskell-build-time dependency on cabal-install
(for `cabal configure`) and the Cabal API (for obtaining the configured
dependency versions and generating the macros).
Any cabal version since Debian Squeeze is supported.

Note that our use of Cabal does not imply any downloading of dependencies
at build time, hermetic builds are unaffected by this change.

For developers we now require hlint >= 1.8.60, to make use of its
--cpp-file option.
However, hlint >= 1.9.12 is recommended since for
hlint >= 1.8.58 && < 1.9.12 the --utf8 flag is non-functional
(see https://github.com/ndmitchell/hlint/issues/96); this can be worked
though by using the equivalent `--encoding=UTF-8` flag.
Remove check for PARALLEL3

This uses the new MIN_VERSION_* infrastructure now.
NEWS: Mention Cabal version enforcement in build system

Also mention the added dependency on Cabal and cabal-install.