dCache/dcache — pull request #177 by gbehrmann (2013-09-11 19:32:22+00:00)
Title: srm: Refactor persistence architecture

Addresses several issues related to persistence of SRM requests:

- Non-strict layering of JobStorage: Jobs could be loaded either from
a JobStorage or through static methods in the Job class. The latter
would canonicalize job instances and implement sharing throughout
a terracotta cluster and it was thus essential that the Job methods
were used. Yet, in several places the JobStorage would be accessed
directly.

This problem is resolved by implementing the shared memory cache and
canonicalization code through JobStorage decorators, thus enforcing
strict layering.

- Use of non-bulk loading of jobs during startup. Since the static
methods of the Job class only supported loading one job at a time,
restarting an SRM would be very slow if the database contained
many non-final jobs.

This is addressed by relying on bulk operations instead. Since
shared memory caching and canonicalization are now implemented as
decorators to JobStorage, bulk operations are now "safe" to use as
they are intercepted and processed by the decorators too.

- Several pieces of code would cast JobStorage to DatabaseJobStorage
to perform special queries, e.g. SrmReleaseFiles. This lead to
several problems: If database persistence is disabled or configured
to only store requests in a final-state, the type cast would not
be possible and thus said functionality would no longer work. The
cast also bypassed the shared memory cache and canonicalization,
and finally the data read from the database may be stale as updates
are written asynchronously.

This is addressed by extending the JobStorage interface with
an operation to fetch non-final jobs, thus allowing these jobs
to be obtained without casting the JobStorage instance and without
bypassing the canonicalization and caching layers.

- During restart, only unscheduled pending jobs would be loaded back
from the database. Thus code that relied on being able to fetch
active jobs from the caching layer would not work after a restart.
In particular the SRM_FILE_BUSY logic would no longer detect busy
files after a restart.

This is addressed by populating the cache with all non-final jobs
during restart. As before, only unscheduled pending jobs are scheduled
for further execution - this can certainly be improved, however
for the purpose of implementing SRM_FILE_BUSY and for properly
releasing TURLs at the end of the transfer, the current logic
suffices.

- Canonicalization relied on a WeakHashMap, which meant that the particular
instance of the request ID (a Long) was significant for the caching. If
that particular Long would no longer have a reference, the entry
would be released.

This is addressed by using a map with weak values created by Guava. This
also means that in a subsequent patch we can replace all those Long IDs
with long.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Paul Millar
Patch: http://rb.dcache.org/r/5929/
(cherry picked from commit 9650be1f8ee50876a58fc116c1ad258b240ee274)