webadmin: fix jquery and Co. with new wicket

wicket 6.9.x comes with own jquery library.
remove the old version. Update pages to use
included version. Update page rendering to
include required scripts.

This change fixes regressions introduced by
migrating to wicket 6.9.1 [55f703e].

Acked-by: Albert Rossi
Target: master, 2.7
Require-book: no
Require-notes: no
webadmin: sort domain entries on CellAdmin page

On a installation with 600 domains it's hard to find
required Domain/Cell.

Acked-by: Paul Millar
Target: master, 2.7
Require-book: no
Require-notes: yes
libs: update jpnfs library

update to a bugfix relase of NFS code:

this fixes enforce security flavor in the export file.

Example entry in export file:

/some/export/path *(rw,sec=krb5)

Changelog for jpnfs-0.5.2..jpnfs-0.5.5
    * [178bddd] nfs: added support for sec flavor in export file
    * [91f4cba] vfs: check security flavor used by client
    * [2cd3029] nfsv3: reply correct auth flavor of mount
    * [ca91195] nfs: ignore and log export entries with unsupported options
    * [be6ff36] junit: refactor v4 test to reduce code duplication
    * [485b6a4] nfsv41: fix open stateid leak

Acked-by: Paul Millar
Target: master, 2.7, 2.6
Require-book: yes
Require-notes: yes
gplazma-nis: plugin should fail on invalid mapping

in set up like:

map      sufficient nis
map      sufficient  authzdb

nis plugin will always stop processing even if mapping is not found.

Ticket: #7946
Acked-by: Paul Millar
Target: trunk, 2.6, 2.7
Require-book: no
Require-notes: yes
ldap: fail if no mapping, make plugin less noisy

The ldap plugin provides map, session and identity functionality.  As
a map and session plugin, the ldap plugin currently always succeeds.
This prevents an admin from making the ldap plugin requisite.

The plugin also logs at error level.  If the admin configures the
plugin as optional then some failures are expected.  For this reason,
the patch moves the logging down the debug level.

FOR RELEASE NOTES:

Currently, the ldap gPlazma plugin never fails in the map- or session-
phases of a login.  With this change, the plugin will fail if no
username is supplied or if there is no mapping for this user.  Sites
should check their gPlazma configuration if they use the ldap plugin.

Target: master
Request: 2.7
Requires-notes: yes
Requires-book: yes
Patch: http://rb.dcache.org/r/5937
Acked-by: Tigran Mkrtchyan
srm: provide better logging for internal errors

The SRM builds an object for each user request that is responsible
for generating the response.  If there is a bug in that object's
class then an InvocationTargetException is thrown.  This is logged
with the stack-trace from the ITE along with the message from the
underlying RuntimeException.

There are two problems: first, the stack-trace is from the ITE, not
the RuntimeException (therefore we don't know where the bug is);
second, the message is logged.  There are several RuntimeExceptions
that have null message, therefore seeing 'null' means we don't know
what went wrong.

This patch fixes these problems.  If the captured problem is an ITE
then the stack-trace of underlying problem reported, not that of the
ITE.  Also, the logged message includes that the problem is due to
an ITE and reports the triggering Exception's toString.

The patch also alters the message send back to the user to simply
"internal error".  Previously the exception's error message was
reported, but I believe that almost all users can do nothing with
this information (it will only confuse them) and the admin should
check the log files when investigating a problem.

Target: master
Request: 2.7
Request: 2.6
Request: 2.2
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7986
Patch: http://rb.dcache.org/r/5936/
Acked-by: Karsten Schwank
srm: Fix and clean up database update logic

Addresses several performance issues in SRM database update
handling.

The logic to write requests to the database suffered from several
problems:

- The request read lock was held while writing the data to the DB.
  Due to a workaround for a Terracotta (which I don't date remove
  even if we don't want to support Terracotta), the read lock is
  actually an exclusive lock. Thus while interacting with the
  database, any operation that requires the lock will block. Thus
  a slow database can indeed have a noticable impact on the SRM.

- When a request is modified several times and thus several saves
  are issued for the same request, each request causes a new
  task for updating the DB to be inserted in the queue. Without
  synchronization, executing two of them simultaneously would
  lead to failures (the task attempts to perform an upsert and
  without an exclusive lock it would be wrong). The code happens
  to work by coincidence due to aforementioned workaround that
  turns the read lock into an eclusive lock.

- The competing update tasks will however block on each other
  (we observed this at NDGF) and thus fewer threads than configured
  will actively write data to the DB.

- Since an update task stored the request state at the time of the
  save (as opposed to the state at the point when save was called),
  repeated update tasks for the same request may end up writing
  the same data to the database - this is wasteful.

  It can also be debated whether the update task should rather
  store the state as it was when save was called, but fixing that
  requires considerable changes in the code, and I don't want to
  do that on stable branches.

- The update task uses several small transactions rather than simply
  execute all statement in a single transaction (which would be
  faster).

- The update task doesn't make use of prepared statement batching
  when writing job history, even though this is an obvious
  and simple performance improvement.

This patch addresses these issues. Rather than blindly inserting a
new update task, a map is used to keep track of requests that have
already been queued for a DB update. Subsequent save attempts will
not create another update task. Thus only a single update task
at a time will work on a request, and it has the added benefit that
during periods of high load, several save operations can be
collapsed.

The update task itself is rewritten to use a single transaction,
to batch job history inserts and protocol table inserts, and to
release the request lock while executing DB statements.

Target: trunk
Request: 2.7
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5900/
srm: Optimize Job loading logic

Addresses performance issues in code to read SRM requests from the
SRM database.

The Job class has a static method for retrieving Jobs by id. The method
is unaware of the Job type though, and thus has to use trial and error
to find the right JobStorage from which to load the request.

This patch refines the logic such that the method makes use of a class
parameter to reduce the number of JobStorages to probe.

A subsequent patch will refine callers of the above method to supply
a more narrow type than is currently the case.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5917/
srm: Fix cut'n'paste error in request history period configuration

Addresses premature garbage collection of put requests.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5930/
srm,gplazma: Use CertPath as x509 cert chain credential in Subject

Addresses that two Subjects cannot be compared using its
equals method if the credential sets contain an certificate
chain.

Addresses a problem in the SRM that causes the gPlazma cache
to never have any hits.

The problem is caused by storing the certificate chain as
an array. Since two arrays are never equal according to
their equals methods, two Subjects cannot be equal either
if they contain X.509 certiticate chains. A consequence of
this bug is that the gPlazma cache in the SRM never finds
a hit as Subjects are never equals to those of a previous
request.

The patch solves the problem by using a CertPath instead.
This class is serializable and implements equals and
hashCode methods. JAAS itself uses CertPath as a public
credential in Subjects.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5941/
fixup!srm,gplazma: Use CertPath as x509 cert chain credential in Subject

Unit tests failed after bc81889861236c245ff60e0a3fe73d2d25c0c787 as
the BC provider would not be loaded.
srm: Generify ContainerRequest, FileRequest and JobStorage

Addresses performance issues in code to load SRM requests from
the SRM database. Also addresses bugs triggered by disabling
database support in the SRM.

The patch generifies a number of SRM classes and interfaces. Besides
removing the need for a lot of type casting, this change allows
generic code in the base classes to know the type of their collaborators.
In particular, this allows a more precise type to be specified when
loading SRM requests back from the database, thus reducing the need
to probe several JobStorages.

The patch removes some dead code to avoid having to update it.

The patch addresses issues in which admin shell commands to cancel
active requests and to query recent requests would fail with a
ClassCastException if database support was disabled. That fix
also addresses issues in which an attempt to cancel requests that
are incompetely stored in the database would cause erroneous
results.

An example of the above issue is

[Gerds-MacBook-Pro.local] (SRM-Gerds-MacBook-Pro) admin > ls completed
java.lang.ClassCastException: org.dcache.srm.scheduler.NoopJobStorage cannot be cast to org.dcache.srm.request.sql.DatabaseRequestStorage

Target: trunk
Require-notes: yes
Require-book: no
srm: Refactor persistence architecture

Addresses several issues related to persistence of SRM requests:

- Non-strict layering of JobStorage: Jobs could be loaded either from
  a JobStorage or through static methods in the Job class. The latter
  would canonicalize job instances and implement sharing throughout
  a terracotta cluster and it was thus essential that the Job methods
  were used. Yet, in several places the JobStorage would be accessed
  directly.

  This problem is resolved by implementing the shared memory cache and
  canonicalization code through JobStorage decorators, thus enforcing
  strict layering.

- Use of non-bulk loading of jobs during startup. Since the static
  methods of the Job class only supported loading one job at a time,
  restarting an SRM would be very slow if the database contained
  many non-final jobs.

  This is addressed by relying on bulk operations instead. Since
  shared memory caching and canonicalization are now implemented as
  decorators to JobStorage, bulk operations are now "safe" to use as
  they are intercepted and processed by the decorators too.

- Several pieces of code would cast JobStorage to DatabaseJobStorage
  to perform special queries, e.g. SrmReleaseFiles. This lead to
  several problems: If database persistence is disabled or configured
  to only store requests in a final-state, the type cast would not
  be possible and thus said functionality would no longer work. The
  cast also bypassed the shared memory cache and canonicalization,
  and finally the data read from the database may be stale as updates
  are written asynchronously.

  This is addressed by extending the JobStorage interface with
  an operation to fetch non-final jobs, thus allowing these jobs
  to be obtained without casting the JobStorage instance and without
  bypassing the canonicalization and caching layers.

- During restart, only unscheduled pending jobs would be loaded back
  from the database. Thus code that relied on being able to fetch
  active jobs from the caching layer would not work after a restart.
  In particular the SRM_FILE_BUSY logic would no longer detect busy
  files after a restart.

  This is addressed by populating the cache with all non-final jobs
  during restart. As before, only unscheduled pending jobs are scheduled
  for further execution - this can certainly be improved, however
  for the purpose of implementing SRM_FILE_BUSY and for properly
  releasing TURLs at the end of the transfer, the current logic
  suffices.

- Canonicalization relied on a WeakHashMap, which meant that the particular
  instance of the request ID (a Long) was significant for the caching. If
  that particular Long would no longer have a reference, the entry
  would be released.

  This is addressed by using a map with weak values created by Guava. This
  also means that in a subsequent patch we can replace all those Long IDs
  with long.

Target: trunk
Request: ?
Require-notes: yes
Require-book: no
spacemanager: Optimize AL/RP injection

Improves latency upon file upload.

Space manager used to intercept PoolAcceptFileMessage,
both to add access latency and retention policy to the
file attributes sent to the pool (so it can choose
precious, cached, and sticky bits) and to send an
update to pnfsmanager to store access latency and
retention policy in chimera.

This patch changes the flow slightly. First of all,
storing access latency and retention policy by sending
an update to pnfsmanager is redundant. Pools in
WriteHandleImpl#registerFileAttributesInNameSpace already
store those attributes in chimera. Second, we can inject
those attributes into the file attributes send to the
pool while doing pool selection. The pool manager will
send back those file attributes to the door. The patch
modifies the doors to pass along those file attributes
to the pool - this is just a minor modification of the
existing logic, as doors already did this exact thing
for storage info.

The result is that for each write we save an additional
queuing phase in space manager, a db transaction and
two reads from the space manager database, and an
AL and RP update in chimera.

Target: trunk
Require-notes: yes
Require-book: no
Interim