ssh2: Honor admin.colors.enable

    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by: Gerd
    Target: master
    Require-book: no
    Require-notes: no

Squashed commit of the following:

commit 8e3db9a3f6cd0c9b8ce6f340f67eb3ecf3651163
Merge: 3887dc4 3e02f94
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 18:33:45 2013 +0200

    Merge color fix branch with current master because not all changes got committed with the last push.

    Sorry.

    ssh2: Honor admin.colors.enable
    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by: Gerd
    Target: master
    Require-book: no
    Require-notes: no

commit 3887dc446e7d388659ffb651e5a0a9e1db67f868
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 14:15:39 2013 +0200

    ssh2: Honor admin.colors.enable

    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by: Gerd
    Target: master
    Require-book: no
    Require-notes: no

commit 3e02f94eebb02bddf81441a9dd7ac41344577e74
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 14:15:39 2013 +0200

    Change parameter to no type definition and switch line to above cell related properties

commit 89bd221adafba600ce35c5f103f65a6736c408e6
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 12:22:09 2013 +0200

    Remove line that sets colors from environment param and not from within dCache

commit 0def3e296a3018c173062457b7256a4ac207ef30
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 11:41:13 2013 +0200

    ssh2: Honor admin.colors.enable

    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by:
    Target: trunk
    Require-book: no
    Require-notes: no
Merge branch 'master' of github.com:dCache/dcache
Merge branch 'master' of github.com:dCache/dcache
Merge remote-tracking branch 'upstream/master'
boot: print reasonable stacktrace for bugs

A recent patch aimed to remove the number of times a stack-trace is
printed if a bug is discovered during start-up.

Unfortunately, the patch resulted in only the outer-most (most recent)
RuntimeException being shown.  We wrap RuntimeExceptions inside
a RuntimeException to provide context-specific information (e.g., a
line number of a file).  Therefore, the most recent stack-trace isn't
interesting.

This patch fixes master so that the root cause is printed instead.

Target: master
Patch: http://rb.dcache.org/r/5958/
Acked-by: Albert Rossi
Requires-notes: no
Requires-book: no
srm: Refactor persistence architecture

Addresses several issues related to persistence of SRM requests:

- Non-strict layering of JobStorage: Jobs could be loaded either from
  a JobStorage or through static methods in the Job class. The latter
  would canonicalize job instances and implement sharing throughout
  a terracotta cluster and it was thus essential that the Job methods
  were used. Yet, in several places the JobStorage would be accessed
  directly.

  This problem is resolved by implementing the shared memory cache and
  canonicalization code through JobStorage decorators, thus enforcing
  strict layering.

- Use of non-bulk loading of jobs during startup. Since the static
  methods of the Job class only supported loading one job at a time,
  restarting an SRM would be very slow if the database contained
  many non-final jobs.

  This is addressed by relying on bulk operations instead. Since
  shared memory caching and canonicalization are now implemented as
  decorators to JobStorage, bulk operations are now "safe" to use as
  they are intercepted and processed by the decorators too.

- Several pieces of code would cast JobStorage to DatabaseJobStorage
  to perform special queries, e.g. SrmReleaseFiles. This lead to
  several problems: If database persistence is disabled or configured
  to only store requests in a final-state, the type cast would not
  be possible and thus said functionality would no longer work. The
  cast also bypassed the shared memory cache and canonicalization,
  and finally the data read from the database may be stale as updates
  are written asynchronously.

  This is addressed by extending the JobStorage interface with
  an operation to fetch non-final jobs, thus allowing these jobs
  to be obtained without casting the JobStorage instance and without
  bypassing the canonicalization and caching layers.

- During restart, only unscheduled pending jobs would be loaded back
  from the database. Thus code that relied on being able to fetch
  active jobs from the caching layer would not work after a restart.
  In particular the SRM_FILE_BUSY logic would no longer detect busy
  files after a restart.

  This is addressed by populating the cache with all non-final jobs
  during restart. As before, only unscheduled pending jobs are scheduled
  for further execution - this can certainly be improved, however
  for the purpose of implementing SRM_FILE_BUSY and for properly
  releasing TURLs at the end of the transfer, the current logic
  suffices.

- Canonicalization relied on a WeakHashMap, which meant that the particular
  instance of the request ID (a Long) was significant for the caching. If
  that particular Long would no longer have a reference, the entry
  would be released.

  This is addressed by using a map with weak values created by Guava. This
  also means that in a subsequent patch we can replace all those Long IDs
  with long.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5929/
srm: Make Scheduler and ModifiableQueue type aware

Addresses the problem that ModifiableQueue restores jobs without
knowing their type, thus incurring extra overhead as several
job storages have to be consulted.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5944/
srm: Change job ID type from Long to long

A recent patch replaced the canonicalization code such that it no
longer relies on a reference to a particular instance of Long. Thus
we can now use long rather than Long to represent required job IDs.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5946/
srm: Implement asynchronous save as a JobStorage decorator

Addresses the problem that DatabaseJobStorage and JdbcConnectionPool
worry about several concerns. This patch separates the queueing
of asynchronous updates from the above two classes and implements
it as a new JobStorage decorator called AsynchronousSaveJobStorage.

The change also reduces the amount of static information in
JdbcConnectionPool.

The queueing is now implemented through an ExecutorService. The rest
of the logic is kept unchanged. A future patch may update
AsynchronousSaveJobStorage to avoid reading stale data from the
database if those jobs are currently queued.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5948/
srm: Replace custom DB connection pool with BoneCP

Among other things, BoneCP does statement caching - something
our custom pool didn't do.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5951/
allow relocation of srmclient scripts

many of the srm scripts point to the generic srm script in
/usr/sbin/srm without using considering $SRM_PATH. This causes
problems if the scripts are installed in different locations than the
ones defined in the RPM.

This patch changes the default location of the generic srm script from
/usr/sbin/srm to ${SRM_PATH}/sbin which is /usr/share/srm/sbin by
default and changes all other scripts to use "${SRM_PATH}/sbin/srm",
thus allowing relocation of the scripts.

Ticket: http://rt.dcache.org/Ticket/Display.html?id=7989
Acked-by:
Target: trunk
Require-book: no
Require-notes: yes
Deprecate a few old properties

The patch fixes a few inconsistencies spotted by Antje.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Antje Petersen <antje.petersen@desy.de>
Patch: http://rb.dcache.org/r/5963/
Replace db.schema.auto with dcache.db.schema.auto

Thanks to Antje for spotting this inconsistency.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: yes
Acked-by: Antje Petersen <antje.petersen@desy.de>
Patch: http://rb.dcache.org/r/5972/
admin: fail commands when user supplies an option that is unknown

The current behaviour for both annotated and ac_ commands is that
any options that are unknown to the command are simply ignored.

This behaviour can be dangerous; for example, the migration
module's move and copy commands accept a 'pnfsid' option to
limit the scope of the operation.  Multiple PNFS-IDs may be
specified as a comma-separated list; however, this leads to
the potential for invoking a move with the option:

     -pnfsids=<id1>,<id2>,...

The migration module will silently ignore the unknown 'pnfsids'
option and move or copy all files on the pool.

More generally, if there is an unknown option then the user is
intending some behaviour that the command will not honour:
irrespective of how dangerous this is, it can be frustrating
to diagnose why an apparently correct invocation isn't doing
what is intended.

This patch fixes this problem for annotated commands only.
This is sufficient to fix the specific issue mentioned in the
ticket.  Solving this for non-annotated commands might be
possible, but I believe only with considerable effort.  Such
effort would be better spent moving those commands to use the
new annotation-based command format.

Target: trunk
Request: 2.7
Request: 2.6
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7991
Patch: http://rb.dcache.org/r/5957/
Acked-by: Albert Rossi
webadmin: wait untill billing cell is ready

StandardBillingService should wait untill billing cell is ready

Ticket: #7981
Acked-by: Albert Rossi
Acked-By: Paul Millar
Target: 2.6, 2.7, master

Require-book: no
Require-notes: no
Merge remote-tracking branch 'upstream/master'
Change ssh admin properties documentation, historyPath
srm: Fix regression in srmCopy

CopyRequest and CopyFileRequest are both processed by the
same SRM scheduler. Thus the scheduler must be configured
for a common base class of these two jobs.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5996/
srm: Do not use SQLException unless its about JDBC

SQLException is the main exception used by JDBC code. The SRM in many
places declares SQLException in code completely unrelated to JDBC, in
code that doesn't even throw SQLException, or uses it to propagate
non-JDBC errors.

This patch addresses this problem, mostly by removing the declaration
of SQLException in the throws clause of many methods. In some cases
SQLException is replaced by another exception, eg IOException or
SRMException.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5953/
srm: Use Spring JDBC support

Spring has extensive support for simplifying JDBC code. This patch updates
the srm to make use of Spring's JDBC support.

At the core of the patch is Spring's JdbcTemplate that provides methods
for common JDBC patterns. Given that the existing code was Connection
oriented, most uses of JdbcTemplate is for the ConnectionCallback pattern:
It encapsulates the acquisition, use and release of a Connection. There
are however plenty of other patterns, some of which are used in the patch.
A future patch may update more of the SRM code to make use of the higher
level patterns.

The patch also replaces most uses of SQLException. SQLExcepton is a catch
all JDBC exception. It has the problem that one has to inspect a vendor
specific error code to get more detailed information about the failure.
More over, a lot of SRM code is not JDBC specific, yet used the JDBC
exception. Eg JobStorage is not limited to JDBC, yet all methods threw
SQLException.

Spring provides the DataAccessException as a replacement for SQLException.
DataAccessException is a base class for an entire hierarchy of data
access related exceptions. Spring translates vendor specific SQL error
codes to an appropriate DataAccessException subclass. The translation
happens automatically when JdbcTemplate is used.

JdbcConnectionPool is now a factory of JdbcTemplate and TransactionTemplate.
It no longer provides methods for acquiring and releasing connections.

A consequence of the patch is that many errors that were previously caught
and ignored are now propagated.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: http://rb.dcache.org/r/5954/