system-test: Fix authzdb generation when htpasswd is not installed

Otherwise gPlazma will fail due to the missing configuration file.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5927/
srm: Optimize Job loading logic

Addresses performance issues in code to read SRM requests from the
SRM database.

The Job class has a static method for retrieving Jobs by id. The method
is unaware of the Job type though, and thus has to use trial and error
to find the right JobStorage from which to load the request.

This patch refines the logic such that the method makes use of a class
parameter to reduce the number of JobStorages to probe.

A subsequent patch will refine callers of the above method to supply
a more narrow type than is currently the case.

Target: trunk
Require-notes: yes
Require-book: no
srm: Generify ContainerRequest, FileRequest and JobStorage

Addresses performance issues in code to load SRM requests from
the SRM database. Also addresses bugs triggered by disabling
database support in the SRM.

The patch generifies a number of SRM classes and interfaces. Besides
removing the need for a lot of type casting, this change allows
generic code in the base classes to know the type of their collaborators.
In particular, this allows a more precise type to be specified when
loading SRM requests back from the database, thus reducing the need
to probe several JobStorages.

The patch removes some dead code to avoid having to update it.

The patch addresses issues in which admin shell commands to cancel
active requests and to query recent requests would fail with a
ClassCastException if database support was disabled. That fix
also addresses issues in which an attempt to cancel requests that
are incompetely stored in the database would cause erroneous
results.

An example of the above issue is

[Gerds-MacBook-Pro.local] (SRM-Gerds-MacBook-Pro) admin > ls completed
java.lang.ClassCastException: org.dcache.srm.scheduler.NoopJobStorage cannot be cast to org.dcache.srm.request.sql.DatabaseRequestStorage

Target: trunk
Require-notes: yes
Require-book: no
srm: Refactor persistence architecture

Addresses several issues related to persistence of SRM requests:

- Non-strict layering of JobStorage: Jobs could be loaded either from
  a JobStorage or through static methods in the Job class. The latter
  would canonicalize job instances and implement sharing throughout
  a terracotta cluster and it was thus essential that the Job methods
  were used. Yet, in several places the JobStorage would be accessed
  directly.

  This problem is resolved by implementing the shared memory cache and
  canonicalization code through JobStorage decorators, thus enforcing
  strict layering.

- Use of non-bulk loading of jobs during startup. Since the static
  methods of the Job class only supported loading one job at a time,
  restarting an SRM would be very slow if the database contained
  many non-final jobs.

  This is addressed by relying on bulk operations instead. Since
  shared memory caching and canonicalization are now implemented as
  decorators to JobStorage, bulk operations are now "safe" to use as
  they are intercepted and processed by the decorators too.

- Several pieces of code would cast JobStorage to DatabaseJobStorage
  to perform special queries, e.g. SrmReleaseFiles. This lead to
  several problems: If database persistence is disabled or configured
  to only store requests in a final-state, the type cast would not
  be possible and thus said functionality would no longer work. The
  cast also bypassed the shared memory cache and canonicalization,
  and finally the data read from the database may be stale as updates
  are written asynchronously.

  This is addressed by extending the JobStorage interface with
  an operation to fetch non-final jobs, thus allowing these jobs
  to be obtained without casting the JobStorage instance and without
  bypassing the canonicalization and caching layers.

- During restart, only unscheduled pending jobs would be loaded back
  from the database. Thus code that relied on being able to fetch
  active jobs from the caching layer would not work after a restart.
  In particular the SRM_FILE_BUSY logic would no longer detect busy
  files after a restart.

  This is addressed by populating the cache with all non-final jobs
  during restart. As before, only unscheduled pending jobs are scheduled
  for further execution - this can certainly be improved, however
  for the purpose of implementing SRM_FILE_BUSY and for properly
  releasing TURLs at the end of the transfer, the current logic
  suffices.

Target: trunk
Request: ?
Require-notes: yes
Require-book: no