chimera: Optimized creation time column changeset

After extensive tests, I found a way to significantly improve the time it takes
to add the creation time column.

The key problem is not adding the column but to update every row in the
t_inodes table to populate the new column. Due to the MVCC nature of postgresql
tables, updating a row doesn't replace it - instead a new copy of the row is
added while the existing row is kept until the transaction within which the
update executes is committed. Since every row in the entire table is updated
in a single transaction, the table effectively doubles in size during this
process. In general, when a row is updated, any index pointing to the row needs
to be updated too. This was the source of the high amount of I/O observed on
our postgresql instance.

Postgresql has a trick to reduce the cost of updates: Heap-only tuples (HOT)
allow the updated row to be stored within the same logical block as the
existing row. In such cases the index doesn't need to be updated, as the index
really points to blocks, not individual rows. HOT updates however require that
there is enough free space in blocks to accomodate additional copies of rows.
During regular operation, deleted rows leave such free space. Also updated rows
once the transaction is committed leave the old copy free to be reclaimed.

In this particular update we however update every record in the entire table
within a single transaction. This means that to avoid expensive index updates,
we need enough free space in blocks to duplicate every row. This is exactly
what this patch achieves: It changes the fillfactor (percentage of a block
filled with regular inserts, leaving the rest available for HOT updates) to
45%, leaving plenty of room for adding an additional copy of every row. After
the update the fillfactor is raised to 75% to avoid wasting too much disk space
in the future.  The default fillfactor in postgresql is 100%, which is fine for
mostly static data, but not fine if rows are updated frequently (we update
inodes frequently due to atime updates and due to link count changes for
directories). The addition of the new column is altered too: We now add the
non-null constraint with a computed default value right away, forcing a table
rewrite (which will use the new fill factor). Once the transaction is
committed, old copies of rows can be reallocated, so there will again be 55%
free space in the table.  Note that after raising the fillfactor to 75%, we do
*not* rewite the table to reclaim the space. Postgresql will use the space when
creating new inodes, but an admin can manually reclaim the space by running
vacuum full on t_inodes (only recommended with Postgresql 9.0 or higher - use
cluster on older versions). Unless tight on disk space, this should not be
necessary.

The difference of this change on our test system is tremendous (similar hardware
to our production system, except significantly less RAM). The old changeset took
approximately 41 hours to complete (for around 48 million inodes - the same
changeset took approx 5 hours on our production hardware). The updated changesets
complete in approx. 40 minutes (half of that is for adding the column; the other
half for update the creation time).

The changeset has been written such that it will not run if the column was already
added to t_inodes (as is the case at NDGF).

Target: trunk
Request: 2.8
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6530/
Include arguments and default values in admin command help output

Also renames a couple of annotation fields to better reflect their
meaning.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6524/
Use annotated command syntax for most shared commands

The patch updates most admin shell commands shared by most services
to the annotated command syntax. This provides nicer help output and
avoid that services like pin manager and space manager show a mixture
of old and new help styles.

The command scanner had to be extended to supported inherited commands.

The help command has been turned into a real command, which means
it will appear in the help output and has a help page too. To avoid
that it gets included multiple times due to being inherited multiple
times, I made the help command optional in the CommandInterpreter.
CellAdapter and UserAmdinShell enable it.

The save and reload commands are now only loaded in services that
actually support these commands.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6531/
system-test: Add utility to start HSQLDB Swing Console

Simplifies manual inspection of the database.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6533/
chimera-nfs: add bringonline command to ".(fset)"

This patch depends on 6409, but also on current trunk/master being fixed (see testing below).  Currently, these changes have not been tested because of the bug blocking their operation.

Adds an operation type to the ".(fset)" dot command to be able to bring a file on line from an archived state.  As with the ".(get)()(locality)" operation, this requires a file system interface in Chimera which is also dCache aware, so that calls can be made to the PoolManager and PinManager.

The following changes have been made:

1.  FileSystemProvider API extended with two methods, pin() and unpin().
2.  PSET inode slightly refactored, with additional call to the filesystem to set pinned state.
3.  PinManager cell endpoint added to DCacheAware filesystem, in order to implement the two added methods.
4.  Relevant timeout properties for the PinManager instance added.

TESTING:

on testbed, using 4.1.  Pinning and unpinning, along with lifetime, works as it should.

Target: master
Request: 2.8
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/6410
Require-notes: yes
Require-book: yes
Depends-on: http://rb.dcache.org/r/6409

BOOK:
RELEASE NOTES:

A new command has been added to the NFS dot-command set to allow users to pin and unpin files.  Files which are not on disk will be brought on line.

The command takes the following form:

            touch ".(fset)(filename)(pin|stage|bringonline)(duration)([SECONDS|MINUTES|HOURS|DAYS])"

The variants of the third argument are equivalent in effect.

The last argument is optional and defaults to SECONDS.

A duration value of 0 will unpin the file.  The command does not allow the user to pin the file indefinitely (the duration value must be a positive integer).

Three new properties associated with the configuration of the pinmanager have been added to nfs.properties:

nfs.service.pinmanager=${dcache.service.pinmanager}
nfs.service.pinmanager.timeout=300000
(one-of?MILLISECONDS|SECONDS|MINUTES|HOURS|DAYS)nfs.service.pinmanager.timeout.unit=MILLISECONDS