ssh2: Honor admin.colors.enable

    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by: Gerd
    Target: master
    Require-book: no
    Require-notes: no

Squashed commit of the following:

commit 8e3db9a3f6cd0c9b8ce6f340f67eb3ecf3651163
Merge: 3887dc4 3e02f94
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 18:33:45 2013 +0200

    Merge color fix branch with current master because not all changes got committed with the last push.

    Sorry.

    ssh2: Honor admin.colors.enable
    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by: Gerd
    Target: master
    Require-book: no
    Require-notes: no

commit 3887dc446e7d388659ffb651e5a0a9e1db67f868
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 14:15:39 2013 +0200

    ssh2: Honor admin.colors.enable

    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by: Gerd
    Target: master
    Require-book: no
    Require-notes: no

commit 3e02f94eebb02bddf81441a9dd7ac41344577e74
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 14:15:39 2013 +0200

    Change parameter to no type definition and switch line to above cell related properties

commit 89bd221adafba600ce35c5f103f65a6736c408e6
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 12:22:09 2013 +0200

    Remove line that sets colors from environment param and not from within dCache

commit 0def3e296a3018c173062457b7256a4ac207ef30
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 11:41:13 2013 +0200

    ssh2: Honor admin.colors.enable

    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by:
    Target: trunk
    Require-book: no
    Require-notes: no
Merge branch 'master' of github.com:dCache/dcache
Merge branch 'master' of github.com:dCache/dcache
Merge remote-tracking branch 'upstream/master'
srm: Generify ContainerRequest, FileRequest and JobStorage

Addresses performance issues in code to load SRM requests from
the SRM database. Also addresses bugs triggered by disabling
database support in the SRM.

The patch generifies a number of SRM classes and interfaces. Besides
removing the need for a lot of type casting, this change allows
generic code in the base classes to know the type of their collaborators.
In particular, this allows a more precise type to be specified when
loading SRM requests back from the database, thus reducing the need
to probe several JobStorages.

The patch removes some dead code to avoid having to update it.

The patch addresses issues in which admin shell commands to cancel
active requests and to query recent requests would fail with a
ClassCastException if database support was disabled. That fix
also addresses issues in which an attempt to cancel requests that
are incompetely stored in the database would cause erroneous
results.

An example of the above issue is

[Gerds-MacBook-Pro.local] (SRM-Gerds-MacBook-Pro) admin > ls completed
java.lang.ClassCastException: org.dcache.srm.scheduler.NoopJobStorage cannot be cast to org.dcache.srm.request.sql.DatabaseRequestStorage

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Request: 2.7
Patch: http://rb.dcache.org/r/5918/
srmclient: Fix RPM prefix

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5928/
spacemanager: Optimize AL/RP injection

Improves latency upon file upload.

Space manager used to intercept PoolAcceptFileMessage,
both to add access latency and retention policy to the
file attributes sent to the pool (so it can choose
precious, cached, and sticky bits) and to send an
update to pnfsmanager to store access latency and
retention policy in chimera.

This patch changes the flow slightly. First of all,
storing access latency and retention policy by sending
an update to pnfsmanager is redundant. Pools in
WriteHandleImpl#registerFileAttributesInNameSpace already
store those attributes in chimera. Second, we can inject
those attributes into the file attributes send to the
pool while doing pool selection. The pool manager will
send back those file attributes to the door. The patch
modifies the doors to pass along those file attributes
to the pool - this is just a minor modification of the
existing logic, as doors already did this exact thing
for storage info.

The result is that for each write we save an additional
queuing phase in space manager, a db transaction and
two reads from the space manager database, and an
AL and RP update in chimera.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5938/
spacemanager: Delete file when cancelling a space reservation

Addresses a problem that may cause inconsistencies between
the files stored in a link group and in spaces of the link
group.

Files are preregistered with space manager before uploading
the file. This reservation can be cancelled before the upload
completes. Such a file will have been authorized for a
particular link group, thus cancelling the reservation should
also break the upload and remove the file.

This patch achieves this by deleting the name space entry
when such a non-final reservation (ie the reservation is
still in RESERVED or TRANSFERRING) is cancelled. This may
happen when the reservation is explicitly cancelled by the
SRM (eg when the client calls srmAbortFiles or the upload
is killed by an srmRm on the SURL), or when the reservation
expires. The file is only deleted if a PNFSID has been
registered, which is the case iff the transfer was authorized
for a link group. The file is deleted by PNFSID, which
guarantees that we do not accidentally delete another file
by the same path.

A consequence of this patch is that if the client does not
finish the upload with a call to srmPutDone, the srm request
will eventually time out and the file will be deleted.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5942/
httpd: fix access to context based pages

The contextHandler was bound to last requested page and
was rejecting access to other values. The internal logic
is change to bound only for 'named' contexts.

RELEASE NOTES:
fixed access to active transfers html and txt pages

Ticket: #7975
Acked-by: Albert Rossi
Target: master, 2.7, 2.6
Require-book: no
Require-notes: yes
boot: print reasonable stacktrace for bugs

A recent patch aimed to remove the number of times a stack-trace is
printed if a bug is discovered during start-up.

Unfortunately, the patch resulted in only the outer-most (most recent)
RuntimeException being shown.  We wrap RuntimeExceptions inside
a RuntimeException to provide context-specific information (e.g., a
line number of a file).  Therefore, the most recent stack-trace isn't
interesting.

This patch fixes master so that the root cause is printed instead.

Target: master
Patch: http://rb.dcache.org/r/5958/
Acked-by: Albert Rossi
Requires-notes: no
Requires-book: no
srm: Refactor persistence architecture

Addresses several issues related to persistence of SRM requests:

- Non-strict layering of JobStorage: Jobs could be loaded either from
  a JobStorage or through static methods in the Job class. The latter
  would canonicalize job instances and implement sharing throughout
  a terracotta cluster and it was thus essential that the Job methods
  were used. Yet, in several places the JobStorage would be accessed
  directly.

  This problem is resolved by implementing the shared memory cache and
  canonicalization code through JobStorage decorators, thus enforcing
  strict layering.

- Use of non-bulk loading of jobs during startup. Since the static
  methods of the Job class only supported loading one job at a time,
  restarting an SRM would be very slow if the database contained
  many non-final jobs.

  This is addressed by relying on bulk operations instead. Since
  shared memory caching and canonicalization are now implemented as
  decorators to JobStorage, bulk operations are now "safe" to use as
  they are intercepted and processed by the decorators too.

- Several pieces of code would cast JobStorage to DatabaseJobStorage
  to perform special queries, e.g. SrmReleaseFiles. This lead to
  several problems: If database persistence is disabled or configured
  to only store requests in a final-state, the type cast would not
  be possible and thus said functionality would no longer work. The
  cast also bypassed the shared memory cache and canonicalization,
  and finally the data read from the database may be stale as updates
  are written asynchronously.

  This is addressed by extending the JobStorage interface with
  an operation to fetch non-final jobs, thus allowing these jobs
  to be obtained without casting the JobStorage instance and without
  bypassing the canonicalization and caching layers.

- During restart, only unscheduled pending jobs would be loaded back
  from the database. Thus code that relied on being able to fetch
  active jobs from the caching layer would not work after a restart.
  In particular the SRM_FILE_BUSY logic would no longer detect busy
  files after a restart.

  This is addressed by populating the cache with all non-final jobs
  during restart. As before, only unscheduled pending jobs are scheduled
  for further execution - this can certainly be improved, however
  for the purpose of implementing SRM_FILE_BUSY and for properly
  releasing TURLs at the end of the transfer, the current logic
  suffices.

- Canonicalization relied on a WeakHashMap, which meant that the particular
  instance of the request ID (a Long) was significant for the caching. If
  that particular Long would no longer have a reference, the entry
  would be released.

  This is addressed by using a map with weak values created by Guava. This
  also means that in a subsequent patch we can replace all those Long IDs
  with long.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5929/
srm: Make Scheduler and ModifiableQueue type aware

Addresses the problem that ModifiableQueue restores jobs without
knowing their type, thus incurring extra overhead as several
job storages have to be consulted.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5944/
srm: Change job ID type from Long to long

A recent patch replaced the canonicalization code such that it no
longer relies on a reference to a particular instance of Long. Thus
we can now use long rather than Long to represent required job IDs.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5946/
srm: Implement asynchronous save as a JobStorage decorator

Addresses the problem that DatabaseJobStorage and JdbcConnectionPool
worry about several concerns. This patch separates the queueing
of asynchronous updates from the above two classes and implements
it as a new JobStorage decorator called AsynchronousSaveJobStorage.

The change also reduces the amount of static information in
JdbcConnectionPool.

The queueing is now implemented through an ExecutorService. The rest
of the logic is kept unchanged. A future patch may update
AsynchronousSaveJobStorage to avoid reading stale data from the
database if those jobs are currently queued.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5948/
srm: Replace custom DB connection pool with BoneCP

Among other things, BoneCP does statement caching - something
our custom pool didn't do.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5951/
allow relocation of srmclient scripts

many of the srm scripts point to the generic srm script in
/usr/sbin/srm without using considering $SRM_PATH. This causes
problems if the scripts are installed in different locations than the
ones defined in the RPM.

This patch changes the default location of the generic srm script from
/usr/sbin/srm to ${SRM_PATH}/sbin which is /usr/share/srm/sbin by
default and changes all other scripts to use "${SRM_PATH}/sbin/srm",
thus allowing relocation of the scripts.

Ticket: http://rt.dcache.org/Ticket/Display.html?id=7989
Acked-by:
Target: trunk
Require-book: no
Require-notes: yes
Deprecate a few old properties

The patch fixes a few inconsistencies spotted by Antje.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Antje Petersen <antje.petersen@desy.de>
Patch: http://rb.dcache.org/r/5963/
Replace db.schema.auto with dcache.db.schema.auto

Thanks to Antje for spotting this inconsistency.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: yes
Acked-by: Antje Petersen <antje.petersen@desy.de>
Patch: http://rb.dcache.org/r/5972/
admin: fail commands when user supplies an option that is unknown

The current behaviour for both annotated and ac_ commands is that
any options that are unknown to the command are simply ignored.

This behaviour can be dangerous; for example, the migration
module's move and copy commands accept a 'pnfsid' option to
limit the scope of the operation.  Multiple PNFS-IDs may be
specified as a comma-separated list; however, this leads to
the potential for invoking a move with the option:

     -pnfsids=<id1>,<id2>,...

The migration module will silently ignore the unknown 'pnfsids'
option and move or copy all files on the pool.

More generally, if there is an unknown option then the user is
intending some behaviour that the command will not honour:
irrespective of how dangerous this is, it can be frustrating
to diagnose why an apparently correct invocation isn't doing
what is intended.

This patch fixes this problem for annotated commands only.
This is sufficient to fix the specific issue mentioned in the
ticket.  Solving this for non-annotated commands might be
possible, but I believe only with considerable effort.  Such
effort would be better spent moving those commands to use the
new annotation-based command format.

Target: trunk
Request: 2.7
Request: 2.6
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7991
Patch: http://rb.dcache.org/r/5957/
Acked-by: Albert Rossi
webadmin: wait untill billing cell is ready

StandardBillingService should wait untill billing cell is ready

Ticket: #7981
Acked-by: Albert Rossi
Acked-By: Paul Millar
Target: 2.6, 2.7, master

Require-book: no
Require-notes: no