spacemanager: Fix forwarding when disabled and add fast forward path

Addresses a bug in which PoolIoFileMessage was not forwarded to pool
manager when spacemanager was disabled.

Addresses a performance issue in which spacemanager congestion caused
by heavy write activity slows down read requests, even though read
requests are simply forwarded by pool manager. The issue is that the
forwarding of messages related to a read request happens on a thread
pool (manager by ThreadManager) that is also used by write requests.
This patch addresses the problem by forwarding such message immediately
rather than scheduling them on a different thread pool.

Note that read requests may still suffer from write congestion as
DoorTransferFinished must be intercepted by space manager and we
cannot distinguish between read and write requests without a
database lookup.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7902
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5886/
Committed: 8270b5e105ead8c8b65881684949245e67f52fce
*dcap, *ftp, ssh1: fix error handling

LoginManager is agnostic to the underlying protocol and delegates
this responsibility to some StreamEngine.  The StreamEngine is
instantated via reflection, which may result in an
InvocationTargetException being thrown, which wraps the underlying
cause of the problem.

There are two problems with the current code: first, if the underlying
problem is an Error (e.g. OutOfMemoryError) then this is not
propagated and dCache will try to continue in this ill-defined state;
second, the error message logged is that of the ITE, which does
not include a description of the underlying problem.

This patch fixes these two problems.

Target: master
Request: 2.6
Request: 2.2
Patch: http://rb.dcache.org/r/5906
Acked-by: Tigran Mkrtchyan
webadmin: sort domain entries on CellAdmin page

On a installation with 600 domains it's hard to find
required Domain/Cell.

Acked-by: Paul Millar
Target: 2.6
Require-book: no
Require-notes: yes
Merge pull request #162 from kofemann/2.6

libs: update jpnfs library
gplazma-nis: plugin should fail on invalid mapping

in set up like:

map      sufficient nis
map      sufficient  authzdb

nis plugin will always stop processing even if mapping is not found.

Ticket: #7946
Acked-by: Paul Millar
Target: trunk, 2.6, 2.7
Require-book: no
Require-notes: yes
Merge pull request #166 from kofemann/iss7946-2.6

Iss7946 2.6
Merge pull request #159 from kofemann/fix-webadmin-sort

Fix webadmin sort
Merge pull request #154 from paulmillar/fix/2.6/log-LoginManager-ITE

Fix/2.6/log login manager ite
Merge pull request #151 from gbehrmann/fix/2.6/rb5886

spacemanager: Fix forwarding when disabled and add fast forward path
srm: provide better logging for internal errors

The SRM builds an object for each user request that is responsible
for generating the response.  If there is a bug in that object's
class then an InvocationTargetException is thrown.  This is logged
with the stack-trace from the ITE along with the message from the
underlying RuntimeException.

There are two problems: first, the stack-trace is from the ITE, not
the RuntimeException (therefore we don't know where the bug is);
second, the message is logged.  There are several RuntimeExceptions
that have null message, therefore seeing 'null' means we don't know
what went wrong.

This patch fixes these problems.  If the captured problem is an ITE
then the stack-trace of underlying problem reported, not that of the
ITE.  Also, the logged message includes that the problem is due to
an ITE and reports the triggering Exception's toString.

The patch also alters the message send back to the user to simply
"internal error".  Previously the exception's error message was
reported, but I believe that almost all users can do nothing with
this information (it will only confuse them) and the admin should
check the log files when investigating a problem.

Target: master
Request: 2.7
Request: 2.6
Request: 2.2
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7986
Patch: http://rb.dcache.org/r/5936/
Acked-by: Karsten Schwank
Requires-notes: no
Requires-book: no
srm: Fix and clean up database update logic

Addresses several performance issues in SRM database update
handling.

The logic to write requests to the database suffered from several
problems:

- The request read lock was held while writing the data to the DB.
  Due to a workaround for a Terracotta (which I don't date remove
  even if we don't want to support Terracotta), the read lock is
  actually an exclusive lock. Thus while interacting with the
  database, any operation that requires the lock will block. Thus
  a slow database can indeed have a noticable impact on the SRM.

- When a request is modified several times and thus several saves
  are issued for the same request, each request causes a new
  task for updating the DB to be inserted in the queue. Without
  synchronization, executing two of them simultaneously would
  lead to failures (the task attempts to perform an upsert and
  without an exclusive lock it would be wrong). The code happens
  to work by coincidence due to aforementioned workaround that
  turns the read lock into an eclusive lock.

- The competing update tasks will however block on each other
  (we observed this at NDGF) and thus fewer threads than configured
  will actively write data to the DB.

- Since an update task stored the request state at the time of the
  save (as opposed to the state at the point when save was called),
  repeated update tasks for the same request may end up writing
  the same data to the database - this is wasteful.

  It can also be debated whether the update task should rather
  store the state as it was when save was called, but fixing that
  requires considerable changes in the code, and I don't want to
  do that on stable branches.

- The update task uses several small transactions rather than simply
  execute all statement in a single transaction (which would be
  faster).

- The update task doesn't make use of prepared statement batching
  when writing job history, even though this is an obvious
  and simple performance improvement.

This patch addresses these issues. Rather than blindly inserting a
new update task, a map is used to keep track of requests that have
already been queued for a DB update. Subsequent save attempts will
not create another update task. Thus only a single update task
at a time will work on a request, and it has the added benefit that
during periods of high load, several save operations can be
collapsed.

The update task itself is rewritten to use a single transaction,
to batch job history inserts and protocol table inserts, and to
release the request lock while executing DB statements.

Target: trunk
Request: 2.7
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5900/
(cherry picked from commit 8ff89b1b7c21c42dc18c5cfcc93907b32bac3307)
srm: Optimize Job loading logic

Addresses performance issues in code to read SRM requests from the
SRM database.

The Job class has a static method for retrieving Jobs by id. The method
is unaware of the Job type though, and thus has to use trial and error
to find the right JobStorage from which to load the request.

This patch refines the logic such that the method makes use of a class
parameter to reduce the number of JobStorages to probe.

A subsequent patch will refine callers of the above method to supply
a more narrow type than is currently the case.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5917/
(cherry picked from commit 406f34bfa022dab24eb2db8300f4fbb130585973)
Merge pull request #168 from gbehrmann/fix/2.6/rb5900

Fix/2.6/rb5900
Merge pull request #170 from gbehrmann/fix/2.6/rb5917

srm: Optimize Job loading logic
[maven-release-plugin] prepare release 2.6.7
[maven-release-plugin] prepare for next development iteration
httpd: fix access to context based pages

The contextHandler was bound to last requested page and
was rejecting access to other values. The internal logic
is change to bound only for 'named' contexts.

RELEASE NOTES:
fixed access to active transfers html and txt pages

Ticket: #7975
Acked-by: Albert Rossi
Target: master, 2.7, 2.6
Require-book: no
Require-notes: yes
Merge pull request #176 from kofemann/iss7975-2.6

Iss7975 2.6
webadmin: wait untill billing cell is ready

StandardBillingService should wait untill billing cell is ready

Ticket: #7981
Acked-by: Paul Millar
Acked-by: Albert Rossi
Target: 2.6, 2.7, master
Require-book: no
Require-notes: no