gplazma-nis: plugin should fail on invalid mapping

in set up like:

map      sufficient nis
map      sufficient  authzdb

nis plugin will always stop processing even if mapping is not found.

Ticket: #7946
Acked-by: Paul Millar
Target: trunk, 2.6, 2.7
Require-book: no
Require-notes: yes
ldap: fail if no mapping, make plugin less noisy

The ldap plugin provides map, session and identity functionality.  As
a map and session plugin, the ldap plugin currently always succeeds.
This prevents an admin from making the ldap plugin requisite.

The plugin also logs at error level.  If the admin configures the
plugin as optional then some failures are expected.  For this reason,
the patch moves the logging down the debug level.

FOR RELEASE NOTES:

Currently, the ldap gPlazma plugin never fails in the map- or session-
phases of a login.  With this change, the plugin will fail if no
username is supplied or if there is no mapping for this user.  Sites
should check their gPlazma configuration if they use the ldap plugin.

Target: master
Request: 2.7
Requires-notes: yes
Requires-book: yes
Patch: http://rb.dcache.org/r/5937
Acked-by: Tigran Mkrtchyan
srm: provide better logging for internal errors

The SRM builds an object for each user request that is responsible
for generating the response.  If there is a bug in that object's
class then an InvocationTargetException is thrown.  This is logged
with the stack-trace from the ITE along with the message from the
underlying RuntimeException.

There are two problems: first, the stack-trace is from the ITE, not
the RuntimeException (therefore we don't know where the bug is);
second, the message is logged.  There are several RuntimeExceptions
that have null message, therefore seeing 'null' means we don't know
what went wrong.

This patch fixes these problems.  If the captured problem is an ITE
then the stack-trace of underlying problem reported, not that of the
ITE.  Also, the logged message includes that the problem is due to
an ITE and reports the triggering Exception's toString.

The patch also alters the message send back to the user to simply
"internal error".  Previously the exception's error message was
reported, but I believe that almost all users can do nothing with
this information (it will only confuse them) and the admin should
check the log files when investigating a problem.

Target: master
Request: 2.7
Request: 2.6
Request: 2.2
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7986
Patch: http://rb.dcache.org/r/5936/
Acked-by: Karsten Schwank
srm: Fix and clean up database update logic

Addresses several performance issues in SRM database update
handling.

The logic to write requests to the database suffered from several
problems:

- The request read lock was held while writing the data to the DB.
  Due to a workaround for a Terracotta (which I don't date remove
  even if we don't want to support Terracotta), the read lock is
  actually an exclusive lock. Thus while interacting with the
  database, any operation that requires the lock will block. Thus
  a slow database can indeed have a noticable impact on the SRM.

- When a request is modified several times and thus several saves
  are issued for the same request, each request causes a new
  task for updating the DB to be inserted in the queue. Without
  synchronization, executing two of them simultaneously would
  lead to failures (the task attempts to perform an upsert and
  without an exclusive lock it would be wrong). The code happens
  to work by coincidence due to aforementioned workaround that
  turns the read lock into an eclusive lock.

- The competing update tasks will however block on each other
  (we observed this at NDGF) and thus fewer threads than configured
  will actively write data to the DB.

- Since an update task stored the request state at the time of the
  save (as opposed to the state at the point when save was called),
  repeated update tasks for the same request may end up writing
  the same data to the database - this is wasteful.

  It can also be debated whether the update task should rather
  store the state as it was when save was called, but fixing that
  requires considerable changes in the code, and I don't want to
  do that on stable branches.

- The update task uses several small transactions rather than simply
  execute all statement in a single transaction (which would be
  faster).

- The update task doesn't make use of prepared statement batching
  when writing job history, even though this is an obvious
  and simple performance improvement.

This patch addresses these issues. Rather than blindly inserting a
new update task, a map is used to keep track of requests that have
already been queued for a DB update. Subsequent save attempts will
not create another update task. Thus only a single update task
at a time will work on a request, and it has the added benefit that
during periods of high load, several save operations can be
collapsed.

The update task itself is rewritten to use a single transaction,
to batch job history inserts and protocol table inserts, and to
release the request lock while executing DB statements.

Target: trunk
Request: 2.7
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5900/
srm: Optimize Job loading logic

Addresses performance issues in code to read SRM requests from the
SRM database.

The Job class has a static method for retrieving Jobs by id. The method
is unaware of the Job type though, and thus has to use trial and error
to find the right JobStorage from which to load the request.

This patch refines the logic such that the method makes use of a class
parameter to reduce the number of JobStorages to probe.

A subsequent patch will refine callers of the above method to supply
a more narrow type than is currently the case.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5917/
srm: Fix cut'n'paste error in request history period configuration

Addresses premature garbage collection of put requests.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5930/
srm,gplazma: Use CertPath as x509 cert chain credential in Subject

Addresses that two Subjects cannot be compared using its
equals method if the credential sets contain an certificate
chain.

Addresses a problem in the SRM that causes the gPlazma cache
to never have any hits.

The problem is caused by storing the certificate chain as
an array. Since two arrays are never equal according to
their equals methods, two Subjects cannot be equal either
if they contain X.509 certiticate chains. A consequence of
this bug is that the gPlazma cache in the SRM never finds
a hit as Subjects are never equals to those of a previous
request.

The patch solves the problem by using a CertPath instead.
This class is serializable and implements equals and
hashCode methods. JAAS itself uses CertPath as a public
credential in Subjects.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5941/
fixup!srm,gplazma: Use CertPath as x509 cert chain credential in Subject

Unit tests failed after bc81889861236c245ff60e0a3fe73d2d25c0c787 as
the BC provider would not be loaded.
fix m2e NPE because of missing versionRange problem

eclipse's m2e plugin has a bug
https://bugs.eclipse.org/bugs/show_bug.cgi?id=347602
that causes it to throw an NPE if the versionRange
field is defined for a plugin.
The versionRange in dcache-core pom.xml was recently
removed with commit 6fe488be16de6a5e1ada58bba99a6f9f6f2c3645
because it caused maven to pull conflicting versions of
the datanucleus-maven-plugin.
This patch re-introduces the verionRange field, but uses
${datanucleus.plugin.version} instead of the open version
range used before.
This fixes the eclipse m2e while only pulling the version
defined in dache-parent.

Ticket:
Acked-by: Al
Acked-by: Paul
Target: master
Require-book: no
Require-notes: no
nfsv41: simplify NfsTransferService and READ/WRITE ops interaction

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no