spacemanager: More fluent API and less business logic in data access layer

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6460/
spacemanager: Add not null and unique constraints to link group name

Since link group name is the only identifier exposed throgh
the UI, this patch marks the field as not null. This requirement
isn't new as the name has always been the only synchronization
point between pool manager and space manager.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6469/
spacemanager: Add automatic and manual purge of perished spaces

Space manager kept spaces indefinitely, even after they had been released or
expired. Due to implicit space reservation, these reservation would add up over
time.  This patch adds automatic deletion of released or expired (perished)
reservations once a certain time has passed since the expiration date. Empty
reservations are purged as soon as they are marked as expired or released.

The patch also adds the 'purge spaces' admin command to purge all perished
spaces immediately, even those that don't have an expiration date. No files are
deleted from dCache and no spaces that are not already released or expired are
purged.

The patch moves the related business logic out of the data access layer. The
fluent API is extended to cope with the new requirements.

Introduces these new properties:

spacemanager.perished-space-purge-delay
spacemanager.perished-space-purge-delay.unit

Introduces this new admin ommand:

purge spaces

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6461/
spacemanager: Use FsPath to represent file system paths

I find using FsPath rather than just String to represent paths
in our name space often makes the code cleaner. This patch
update space manager to use FsPath in this way.

The path removes a number of unused fields from our messages and
thus simplifies the logic in space manager.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6465/
spacemanager: Simplify updateFile method

Alters the logic for updating file records. Rather than
passing the fields to change as arguments to the update method,
the code now simply uses the setter on the File object directly
and saves the result. Besides making the code simpler, it
also allowed a number of update methods to be removed from the
DAO interface - these methods were needed because updateFile
was unable to set fields to null.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6466/
spacemanager: Clear path when file is stored and make path unique

The only purpose of the path column in space manager is to allocate
space to a file before it is actually created in the name space. Once
the file is stored, the PNFS ID identifies the file. At this
point the path is unnecessary bloat in the database, which - if the file
is renamed or moved (as recently happened in Atlas) - may even be
different from the actual path.

This patch changes the use of the path field such that it is always
cleared once the file is marked as stored. The field is marked as a
unique field, which means that lookup code becomes simpler. The need
for explicit indexes disappears (unique fields have an implicit index,
as the combined index on path and state is no longer necessary as the
code can load the record by the unique path and check the state in
the business logic).

More business logic has been moved out of the data access layer, thanks
to the uniquenes of the path.

The error message for duplicate paths (the infamous "Already have 1
record(s)" message has been replaced with a more descriptive message
(that the file is locked by another transfer).

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6467/
spacemanager: Turn state singletons into enums

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6470/
spacemanager: Delete dead code

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6471/
spacemanager,webadmin,info: Only publish permanent space reservations

Space reservations that are not permanent are usually implicit space
reservations. What the admin and the rest of the world are interested
in are the permanent reservations usually created by the admin.

The patch removes the expiration date and lifetime columns from webadmin
as these are no longer relevant.

Also fixed a type cast bug recently introduced when changing the message
format.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6478/
spacemanager: Use HikariCP

To gain experience with an alternative to BoneCP, this patch
switches space manager to HikariCP. I chose space manager because
the configuration parameters for the connection pool where new
in 2.8 and thus they are easy to change now.

Introduces these new properties:

spacemanager.db.connections.max
spacemanager.db.connections.min

Drops these unpublished properties:

spacemanager.db.connections.partition-count
spacemanager.db.connections.max-per-partition
spacemanager.db.connections.min-per-partition

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6475/
spacemanager: Reorder liquibase changesets to make updates faster

If we drop and delete stuff before doing things that trigger
a table rewrite, then upgrading to 2.8 will be faster.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6481/
spacemanager: Make changesets drop indexes conditional

This protects against sites that have made local modifications
to the schema (like NDGF).

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6482/
spacemanager: Replace lifetime column with expirationtime column

Uses a null value to represent that there is no expiration time (infinite
lifetime). Also removes indexes on creationtime columns.

Storing expiration time is much more useful as that's what we typically
search for (has expiration time passed).

Changes the DAO API to simplify updateSpace and thus also enable it to
store null.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6483/
spacemanager: Limit ls files command to transient states by default

Defaulting to listing all files in stored state is a bit dangerous as it is
guaranteed to consume lots of memory on most production systems.

This patch changes the defaults to only list files in allocated and
transferring states. These are likely the files an admin would be interested
in, e.g. when wanting to release a lock manually.

Also fixed some help text.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6484/
spacemanager: Rename release file to purge file

This fits better with the equivalent operation in SRM: srmPurgeFromSpace.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6485/
spacemanager: Make help strings less ambiguous

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6486/
Revert "spacemanager: Make help strings less ambiguous"

This reverts commit 9c35c335822eeb0449786f17f149f8a7ed00ca43.
spacemanager: Make help strings less ambiguous

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6486/
deb: Append timestamp to snapshot builds

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6448/
spacemanager: Fix slow SQL query

On small tables the two queries generate the same plan, but when
testing it on a clone of our production system, the queries produce
significantly different results (despite being equivalent when
the columns involved are marked not null).

The new query produces the same query plan as a left outer join
query, so there is no reason to use an explicit join (which would
be difficult to get into the criterion api).

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6487/