ssh2: Honor admin.colors.enable

    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by: Gerd
    Target: master
    Require-book: no
    Require-notes: no

Squashed commit of the following:

commit 8e3db9a3f6cd0c9b8ce6f340f67eb3ecf3651163
Merge: 3887dc4 3e02f94
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 18:33:45 2013 +0200

    Merge color fix branch with current master because not all changes got committed with the last push.

    Sorry.

    ssh2: Honor admin.colors.enable
    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by: Gerd
    Target: master
    Require-book: no
    Require-notes: no

commit 3887dc446e7d388659ffb651e5a0a9e1db67f868
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 14:15:39 2013 +0200

    ssh2: Honor admin.colors.enable

    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by: Gerd
    Target: master
    Require-book: no
    Require-notes: no

commit 3e02f94eebb02bddf81441a9dd7ac41344577e74
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 14:15:39 2013 +0200

    Change parameter to no type definition and switch line to above cell related properties

commit 89bd221adafba600ce35c5f103f65a6736c408e6
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 12:22:09 2013 +0200

    Remove line that sets colors from environment param and not from within dCache

commit 0def3e296a3018c173062457b7256a4ac207ef30
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 11:41:13 2013 +0200

    ssh2: Honor admin.colors.enable

    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by:
    Target: trunk
    Require-book: no
    Require-notes: no
Merge branch 'master' of github.com:dCache/dcache
Merge branch 'master' of github.com:dCache/dcache
srm: Fix and clean up database update logic

Addresses several performance issues in SRM database update
handling.

The logic to write requests to the database suffered from several
problems:

- The request read lock was held while writing the data to the DB.
  Due to a workaround for a Terracotta (which I don't date remove
  even if we don't want to support Terracotta), the read lock is
  actually an exclusive lock. Thus while interacting with the
  database, any operation that requires the lock will block. Thus
  a slow database can indeed have a noticable impact on the SRM.

- When a request is modified several times and thus several saves
  are issued for the same request, each request causes a new
  task for updating the DB to be inserted in the queue. Without
  synchronization, executing two of them simultaneously would
  lead to failures (the task attempts to perform an upsert and
  without an exclusive lock it would be wrong). The code happens
  to work by coincidence due to aforementioned workaround that
  turns the read lock into an eclusive lock.

- The competing update tasks will however block on each other
  (we observed this at NDGF) and thus fewer threads than configured
  will actively write data to the DB.

- Since an update task stored the request state at the time of the
  save (as opposed to the state at the point when save was called),
  repeated update tasks for the same request may end up writing
  the same data to the database - this is wasteful.

  It can also be debated whether the update task should rather
  store the state as it was when save was called, but fixing that
  requires considerable changes in the code, and I don't want to
  do that on stable branches.

- The update task uses several small transactions rather than simply
  execute all statement in a single transaction (which would be
  faster).

- The update task doesn't make use of prepared statement batching
  when writing job history, even though this is an obvious
  and simple performance improvement.

This patch addresses these issues. Rather than blindly inserting a
new update task, a map is used to keep track of requests that have
already been queued for a DB update. Subsequent save attempts will
not create another update task. Thus only a single update task
at a time will work on a request, and it has the added benefit that
during periods of high load, several save operations can be
collapsed.

The update task itself is rewritten to use a single transaction,
to batch job history inserts and protocol table inserts, and to
release the request lock while executing DB statements.

Target: trunk
Request: 2.7
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5900/
srm: Optimize Job loading logic

Addresses performance issues in code to read SRM requests from the
SRM database.

The Job class has a static method for retrieving Jobs by id. The method
is unaware of the Job type though, and thus has to use trial and error
to find the right JobStorage from which to load the request.

This patch refines the logic such that the method makes use of a class
parameter to reduce the number of JobStorages to probe.

A subsequent patch will refine callers of the above method to supply
a more narrow type than is currently the case.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5917/
srm: Fix cut'n'paste error in request history period configuration

Addresses premature garbage collection of put requests.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5930/
srm,gplazma: Use CertPath as x509 cert chain credential in Subject

Addresses that two Subjects cannot be compared using its
equals method if the credential sets contain an certificate
chain.

Addresses a problem in the SRM that causes the gPlazma cache
to never have any hits.

The problem is caused by storing the certificate chain as
an array. Since two arrays are never equal according to
their equals methods, two Subjects cannot be equal either
if they contain X.509 certiticate chains. A consequence of
this bug is that the gPlazma cache in the SRM never finds
a hit as Subjects are never equals to those of a previous
request.

The patch solves the problem by using a CertPath instead.
This class is serializable and implements equals and
hashCode methods. JAAS itself uses CertPath as a public
credential in Subjects.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5941/
fixup!srm,gplazma: Use CertPath as x509 cert chain credential in Subject

Unit tests failed after bc81889861236c245ff60e0a3fe73d2d25c0c787 as
the BC provider would not be loaded.
fix m2e NPE because of missing versionRange problem

eclipse's m2e plugin has a bug
https://bugs.eclipse.org/bugs/show_bug.cgi?id=347602
that causes it to throw an NPE if the versionRange
field is defined for a plugin.
The versionRange in dcache-core pom.xml was recently
removed with commit 6fe488be16de6a5e1ada58bba99a6f9f6f2c3645
because it caused maven to pull conflicting versions of
the datanucleus-maven-plugin.
This patch re-introduces the verionRange field, but uses
${datanucleus.plugin.version} instead of the open version
range used before.
This fixes the eclipse m2e while only pulling the version
defined in dache-parent.

Ticket:
Acked-by: Al
Acked-by: Paul
Target: master
Require-book: no
Require-notes: no
nfsv41: simplify NfsTransferService and READ/WRITE ops interaction

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
nfsv41: TransferService sholud disable pool no disk errors

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
Merge remote-tracking branch 'upstream/master'
srm: Generify ContainerRequest, FileRequest and JobStorage

Addresses performance issues in code to load SRM requests from
the SRM database. Also addresses bugs triggered by disabling
database support in the SRM.

The patch generifies a number of SRM classes and interfaces. Besides
removing the need for a lot of type casting, this change allows
generic code in the base classes to know the type of their collaborators.
In particular, this allows a more precise type to be specified when
loading SRM requests back from the database, thus reducing the need
to probe several JobStorages.

The patch removes some dead code to avoid having to update it.

The patch addresses issues in which admin shell commands to cancel
active requests and to query recent requests would fail with a
ClassCastException if database support was disabled. That fix
also addresses issues in which an attempt to cancel requests that
are incompetely stored in the database would cause erroneous
results.

An example of the above issue is

[Gerds-MacBook-Pro.local] (SRM-Gerds-MacBook-Pro) admin > ls completed
java.lang.ClassCastException: org.dcache.srm.scheduler.NoopJobStorage cannot be cast to org.dcache.srm.request.sql.DatabaseRequestStorage

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Request: 2.7
Patch: http://rb.dcache.org/r/5918/
srmclient: Fix RPM prefix

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5928/
spacemanager: Optimize AL/RP injection

Improves latency upon file upload.

Space manager used to intercept PoolAcceptFileMessage,
both to add access latency and retention policy to the
file attributes sent to the pool (so it can choose
precious, cached, and sticky bits) and to send an
update to pnfsmanager to store access latency and
retention policy in chimera.

This patch changes the flow slightly. First of all,
storing access latency and retention policy by sending
an update to pnfsmanager is redundant. Pools in
WriteHandleImpl#registerFileAttributesInNameSpace already
store those attributes in chimera. Second, we can inject
those attributes into the file attributes send to the
pool while doing pool selection. The pool manager will
send back those file attributes to the door. The patch
modifies the doors to pass along those file attributes
to the pool - this is just a minor modification of the
existing logic, as doors already did this exact thing
for storage info.

The result is that for each write we save an additional
queuing phase in space manager, a db transaction and
two reads from the space manager database, and an
AL and RP update in chimera.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5938/
spacemanager: Delete file when cancelling a space reservation

Addresses a problem that may cause inconsistencies between
the files stored in a link group and in spaces of the link
group.

Files are preregistered with space manager before uploading
the file. This reservation can be cancelled before the upload
completes. Such a file will have been authorized for a
particular link group, thus cancelling the reservation should
also break the upload and remove the file.

This patch achieves this by deleting the name space entry
when such a non-final reservation (ie the reservation is
still in RESERVED or TRANSFERRING) is cancelled. This may
happen when the reservation is explicitly cancelled by the
SRM (eg when the client calls srmAbortFiles or the upload
is killed by an srmRm on the SURL), or when the reservation
expires. The file is only deleted if a PNFSID has been
registered, which is the case iff the transfer was authorized
for a link group. The file is deleted by PNFSID, which
guarantees that we do not accidentally delete another file
by the same path.

A consequence of this patch is that if the client does not
finish the upload with a call to srmPutDone, the srm request
will eventually time out and the file will be deleted.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5942/
httpd: fix access to context based pages

The contextHandler was bound to last requested page and
was rejecting access to other values. The internal logic
is change to bound only for 'named' contexts.

RELEASE NOTES:
fixed access to active transfers html and txt pages

Ticket: #7975
Acked-by: Albert Rossi
Target: master, 2.7, 2.6
Require-book: no
Require-notes: yes
boot: print reasonable stacktrace for bugs

A recent patch aimed to remove the number of times a stack-trace is
printed if a bug is discovered during start-up.

Unfortunately, the patch resulted in only the outer-most (most recent)
RuntimeException being shown.  We wrap RuntimeExceptions inside
a RuntimeException to provide context-specific information (e.g., a
line number of a file).  Therefore, the most recent stack-trace isn't
interesting.

This patch fixes master so that the root cause is printed instead.

Target: master
Patch: http://rb.dcache.org/r/5958/
Acked-by: Albert Rossi
Requires-notes: no
Requires-book: no
srm: Refactor persistence architecture

Addresses several issues related to persistence of SRM requests:

- Non-strict layering of JobStorage: Jobs could be loaded either from
  a JobStorage or through static methods in the Job class. The latter
  would canonicalize job instances and implement sharing throughout
  a terracotta cluster and it was thus essential that the Job methods
  were used. Yet, in several places the JobStorage would be accessed
  directly.

  This problem is resolved by implementing the shared memory cache and
  canonicalization code through JobStorage decorators, thus enforcing
  strict layering.

- Use of non-bulk loading of jobs during startup. Since the static
  methods of the Job class only supported loading one job at a time,
  restarting an SRM would be very slow if the database contained
  many non-final jobs.

  This is addressed by relying on bulk operations instead. Since
  shared memory caching and canonicalization are now implemented as
  decorators to JobStorage, bulk operations are now "safe" to use as
  they are intercepted and processed by the decorators too.

- Several pieces of code would cast JobStorage to DatabaseJobStorage
  to perform special queries, e.g. SrmReleaseFiles. This lead to
  several problems: If database persistence is disabled or configured
  to only store requests in a final-state, the type cast would not
  be possible and thus said functionality would no longer work. The
  cast also bypassed the shared memory cache and canonicalization,
  and finally the data read from the database may be stale as updates
  are written asynchronously.

  This is addressed by extending the JobStorage interface with
  an operation to fetch non-final jobs, thus allowing these jobs
  to be obtained without casting the JobStorage instance and without
  bypassing the canonicalization and caching layers.

- During restart, only unscheduled pending jobs would be loaded back
  from the database. Thus code that relied on being able to fetch
  active jobs from the caching layer would not work after a restart.
  In particular the SRM_FILE_BUSY logic would no longer detect busy
  files after a restart.

  This is addressed by populating the cache with all non-final jobs
  during restart. As before, only unscheduled pending jobs are scheduled
  for further execution - this can certainly be improved, however
  for the purpose of implementing SRM_FILE_BUSY and for properly
  releasing TURLs at the end of the transfer, the current logic
  suffices.

- Canonicalization relied on a WeakHashMap, which meant that the particular
  instance of the request ID (a Long) was significant for the caching. If
  that particular Long would no longer have a reference, the entry
  would be released.

  This is addressed by using a map with weak values created by Guava. This
  also means that in a subsequent patch we can replace all those Long IDs
  with long.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5929/
srm: Make Scheduler and ModifiableQueue type aware

Addresses the problem that ModifiableQueue restores jobs without
knowing their type, thus incurring extra overhead as several
job storages have to be consulted.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5944/