srm: Generify ContainerRequest, FileRequest and JobStorage

Addresses performance issues in code to load SRM requests from
the SRM database. Also addresses bugs triggered by disabling
database support in the SRM.

The patch generifies a number of SRM classes and interfaces. Besides
removing the need for a lot of type casting, this change allows
generic code in the base classes to know the type of their collaborators.
In particular, this allows a more precise type to be specified when
loading SRM requests back from the database, thus reducing the need
to probe several JobStorages.

The patch removes some dead code to avoid having to update it.

The patch addresses issues in which admin shell commands to cancel
active requests and to query recent requests would fail with a
ClassCastException if database support was disabled. That fix
also addresses issues in which an attempt to cancel requests that
are incompetely stored in the database would cause erroneous
results.

An example of the above issue is

[Gerds-MacBook-Pro.local] (SRM-Gerds-MacBook-Pro) admin > ls completed
java.lang.ClassCastException: org.dcache.srm.scheduler.NoopJobStorage cannot be cast to org.dcache.srm.request.sql.DatabaseRequestStorage

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Request: 2.7
Patch: http://rb.dcache.org/r/5918/
srmclient: Fix RPM prefix

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5928/
spacemanager: Optimize AL/RP injection

Improves latency upon file upload.

Space manager used to intercept PoolAcceptFileMessage,
both to add access latency and retention policy to the
file attributes sent to the pool (so it can choose
precious, cached, and sticky bits) and to send an
update to pnfsmanager to store access latency and
retention policy in chimera.

This patch changes the flow slightly. First of all,
storing access latency and retention policy by sending
an update to pnfsmanager is redundant. Pools in
WriteHandleImpl#registerFileAttributesInNameSpace already
store those attributes in chimera. Second, we can inject
those attributes into the file attributes send to the
pool while doing pool selection. The pool manager will
send back those file attributes to the door. The patch
modifies the doors to pass along those file attributes
to the pool - this is just a minor modification of the
existing logic, as doors already did this exact thing
for storage info.

The result is that for each write we save an additional
queuing phase in space manager, a db transaction and
two reads from the space manager database, and an
AL and RP update in chimera.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5938/
spacemanager: Delete file when cancelling a space reservation

Addresses a problem that may cause inconsistencies between
the files stored in a link group and in spaces of the link
group.

Files are preregistered with space manager before uploading
the file. This reservation can be cancelled before the upload
completes. Such a file will have been authorized for a
particular link group, thus cancelling the reservation should
also break the upload and remove the file.

This patch achieves this by deleting the name space entry
when such a non-final reservation (ie the reservation is
still in RESERVED or TRANSFERRING) is cancelled. This may
happen when the reservation is explicitly cancelled by the
SRM (eg when the client calls srmAbortFiles or the upload
is killed by an srmRm on the SURL), or when the reservation
expires. The file is only deleted if a PNFSID has been
registered, which is the case iff the transfer was authorized
for a link group. The file is deleted by PNFSID, which
guarantees that we do not accidentally delete another file
by the same path.

A consequence of this patch is that if the client does not
finish the upload with a call to srmPutDone, the srm request
will eventually time out and the file will be deleted.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5942/
httpd: fix access to context based pages

The contextHandler was bound to last requested page and
was rejecting access to other values. The internal logic
is change to bound only for 'named' contexts.

RELEASE NOTES:
fixed access to active transfers html and txt pages

Ticket: #7975
Acked-by: Albert Rossi
Target: master, 2.7, 2.6
Require-book: no
Require-notes: yes
boot: print reasonable stacktrace for bugs

A recent patch aimed to remove the number of times a stack-trace is
printed if a bug is discovered during start-up.

Unfortunately, the patch resulted in only the outer-most (most recent)
RuntimeException being shown.  We wrap RuntimeExceptions inside
a RuntimeException to provide context-specific information (e.g., a
line number of a file).  Therefore, the most recent stack-trace isn't
interesting.

This patch fixes master so that the root cause is printed instead.

Target: master
Patch: http://rb.dcache.org/r/5958/
Acked-by: Albert Rossi
Requires-notes: no
Requires-book: no
srm: Refactor persistence architecture

Addresses several issues related to persistence of SRM requests:

- Non-strict layering of JobStorage: Jobs could be loaded either from
  a JobStorage or through static methods in the Job class. The latter
  would canonicalize job instances and implement sharing throughout
  a terracotta cluster and it was thus essential that the Job methods
  were used. Yet, in several places the JobStorage would be accessed
  directly.

  This problem is resolved by implementing the shared memory cache and
  canonicalization code through JobStorage decorators, thus enforcing
  strict layering.

- Use of non-bulk loading of jobs during startup. Since the static
  methods of the Job class only supported loading one job at a time,
  restarting an SRM would be very slow if the database contained
  many non-final jobs.

  This is addressed by relying on bulk operations instead. Since
  shared memory caching and canonicalization are now implemented as
  decorators to JobStorage, bulk operations are now "safe" to use as
  they are intercepted and processed by the decorators too.

- Several pieces of code would cast JobStorage to DatabaseJobStorage
  to perform special queries, e.g. SrmReleaseFiles. This lead to
  several problems: If database persistence is disabled or configured
  to only store requests in a final-state, the type cast would not
  be possible and thus said functionality would no longer work. The
  cast also bypassed the shared memory cache and canonicalization,
  and finally the data read from the database may be stale as updates
  are written asynchronously.

  This is addressed by extending the JobStorage interface with
  an operation to fetch non-final jobs, thus allowing these jobs
  to be obtained without casting the JobStorage instance and without
  bypassing the canonicalization and caching layers.

- During restart, only unscheduled pending jobs would be loaded back
  from the database. Thus code that relied on being able to fetch
  active jobs from the caching layer would not work after a restart.
  In particular the SRM_FILE_BUSY logic would no longer detect busy
  files after a restart.

  This is addressed by populating the cache with all non-final jobs
  during restart. As before, only unscheduled pending jobs are scheduled
  for further execution - this can certainly be improved, however
  for the purpose of implementing SRM_FILE_BUSY and for properly
  releasing TURLs at the end of the transfer, the current logic
  suffices.

- Canonicalization relied on a WeakHashMap, which meant that the particular
  instance of the request ID (a Long) was significant for the caching. If
  that particular Long would no longer have a reference, the entry
  would be released.

  This is addressed by using a map with weak values created by Guava. This
  also means that in a subsequent patch we can replace all those Long IDs
  with long.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5929/
srm: Make Scheduler and ModifiableQueue type aware

Addresses the problem that ModifiableQueue restores jobs without
knowing their type, thus incurring extra overhead as several
job storages have to be consulted.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5944/
srm: Change job ID type from Long to long

A recent patch replaced the canonicalization code such that it no
longer relies on a reference to a particular instance of Long. Thus
we can now use long rather than Long to represent required job IDs.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5946/
srm: Implement asynchronous save as a JobStorage decorator

Addresses the problem that DatabaseJobStorage and JdbcConnectionPool
worry about several concerns. This patch separates the queueing
of asynchronous updates from the above two classes and implements
it as a new JobStorage decorator called AsynchronousSaveJobStorage.

The change also reduces the amount of static information in
JdbcConnectionPool.

The queueing is now implemented through an ExecutorService. The rest
of the logic is kept unchanged. A future patch may update
AsynchronousSaveJobStorage to avoid reading stale data from the
database if those jobs are currently queued.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5948/
srm: Replace custom DB connection pool with BoneCP

Among other things, BoneCP does statement caching - something
our custom pool didn't do.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5951/
allow relocation of srmclient scripts

many of the srm scripts point to the generic srm script in
/usr/sbin/srm without using considering $SRM_PATH. This causes
problems if the scripts are installed in different locations than the
ones defined in the RPM.

This patch changes the default location of the generic srm script from
/usr/sbin/srm to ${SRM_PATH}/sbin which is /usr/share/srm/sbin by
default and changes all other scripts to use "${SRM_PATH}/sbin/srm",
thus allowing relocation of the scripts.

Ticket: http://rt.dcache.org/Ticket/Display.html?id=7989
Acked-by:
Target: trunk
Require-book: no
Require-notes: yes
Merge remote-tracking branch 'upstream/master'