Add LUXID_ARGS to defaults/ganeti

luxid accepts arguments as well.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
design: scheduling will be simple in the first iteration

In order to progress faster towards jobs as processes, the decision on
the final scheduling algorithm will be postponed. For the time being,
the simple one will be used that starts jobs until a fixed maximal
number of running jobs is reached. This postponing also has the
advantage that the final decision can be based on experience with the
characteristics of jobs as processes under realistic load.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Rename enqueueJobs to startJobs

This reflects better what the method actually does. Later,
we will add a job scheduler that will provide a proper enqueue
method.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Move FStat related function to Utils

In this way, the functions to to decide, based on fstat, whether
a file needs to be reloaded can used by other parts as well,
in particular to monitor progress in the job queue.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Add a scheduler to keep track of the job queue

In order to allow informed decissions on when to start a job,
it is necessary for luxid to keep track of the (active part
of the) job queue. Add a scheduler, similar to the config reader,
that does this, but also schedules jobs to be executed. At the
moment this is just the simple mechanism of starting jobs until
a fixed number of running jobs is reached.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Make luxid use the JQScheduler

Make luxid use the job scheduler instead of immediately
starting every received job.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Petr Pudlak <pudlak@google.com>
Differentiate watchers in luxid

luxid has two time-based watcher threads, one for the
configuration, and one for the job queue. To improve readability
of the debug output, make both watcher use a different debug
message when the timer fires.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Jose Lopes <jabolopes@google.com>
fix typo in log message

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Jose Lopes <jabolopes@google.com>
Make JQScheduler handle failure on job starting

Given that luxid (at the moment) connects to masterd for starting
jobs, it may be that this inter-process communication fails. In
this case, just reschedule the jobs instead of killing the scheduler
thread.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Jose Lopes <jabolopes@google.com>
Add a new unit parsing function taking all suffixes binary

In python, when parsing units (like disk sizes) we take all
suffixes (M, G, T) as 1024-based. To be backwards compatible
while moving to job management to luxid, in particular on RAPI,
add such a parsing function in haskell as well.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Add a field-transformer for accepting parser

Add a transformer for numerical fields, to also accept
strings instead of numbers if they can be parsed by the
given parser.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Make disk size a special numerical field

For disk sizes, instead of plain numbers (naming the value in MiB),
also accept expressions with units like 'GiB'.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Support fieldRead in partial params

While from parameters both full and partial versions are generated,
with in the partial version all types mapped to Maybe, the fieldRead
parameter of the field was not wrapped accordingly. So far, that didn't
matter, as it was always Nothing in this case, but for supporting special
parsing functions for memory parameters, this is now needed.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Support size suffixes in minmem/maxmem backed parameters

The backed parameters specifying the minimal/maximal memory
can also be passed as values with suffixes. Support parsing
these values.

Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Hrvoje Ribicic <riba@google.com>
Gluster: Update design document

Anticipate and explain the choices made in the Gluster patch series.
Remove parts about a possible way of supporting userspace access as
it has been implemented otherwise.
bdev: Fix position of DEV_MAP

This rather important dictionary from constants to classes was hiding
between function definitions. The dict cannot go to the top of the file
as the classes haven't been defined there yet, so it's been pushed
to the bottom of the file.

For bonus points, the dict gets an epydoc docstring and a comment, too.
ComputeLDParams: do not spell out disk templates unnecessarily

A large part of the complexity in this function is due to the need
to translate from "template-specific" parameter names to
"template-agnostic" parameter names. This logic is complex and having
complex code for complex logic is okay.

However, for all other cases, the additional complexity is not needed.
Additionally this function would do the wrong thing silently for new,
unhandled disk templates by just not changing the resulting paramters.

This commit replaces the ad-hoc logic with a more generally useful
logic while leaving the DRBD code carefully alone.
PathJoin: improve error message when given one argument

PathJoin fails with an unclear message if only one argument is passed
to it. Calling PathJoin("/foo") causes this exception:

    Error: path joining resulted in different prefix (/foo != /foo)

However, /foo and /foo obviously share prefixes: what this function
really checks is that the resulting path should be inside the first
argument and, in that sense, /foo is not a "real" prefix of /foo.

Given that PathJoin already asserts that it received any arguments
at all, this commit discards the assertion for an error to be raised
if 0 or 1 arguments are received. The semantics don't change, but
the error message is clearer.
FileStorage: move to filesstorage.py

Move the FileStorage class in its own file, together with its helper
functions.
FileStorage: extract file logic to a FileDeviceHelper object

This will allow code reuse for Gluster through composition, rather
than inheritance.