dCache/dcache — pull request #169 by gbehrmann (2013-09-07 08:10:18+00:00)
Title: srm: Fix and clean up database update logic

Addresses several performance issues in SRM database update
handling.

The logic to write requests to the database suffered from several
problems:

- The request read lock was held while writing the data to the DB.
Due to a workaround for a Terracotta (which I don't date remove
even if we don't want to support Terracotta), the read lock is
actually an exclusive lock. Thus while interacting with the
database, any operation that requires the lock will block. Thus
a slow database can indeed have a noticable impact on the SRM.

- When a request is modified several times and thus several saves
are issued for the same request, each request causes a new
task for updating the DB to be inserted in the queue. Without
synchronization, executing two of them simultaneously would
lead to failures (the task attempts to perform an upsert and
without an exclusive lock it would be wrong). The code happens
to work by coincidence due to aforementioned workaround that
turns the read lock into an eclusive lock.

- The competing update tasks will however block on each other
(we observed this at NDGF) and thus fewer threads than configured
will actively write data to the DB.

- Since an update task stored the request state at the time of the
save (as opposed to the state at the point when save was called),
repeated update tasks for the same request may end up writing
the same data to the database - this is wasteful.

It can also be debated whether the update task should rather
store the state as it was when save was called, but fixing that
requires considerable changes in the code, and I don't want to
do that on stable branches.

- The update task uses several small transactions rather than simply
execute all statement in a single transaction (which would be
faster).

- The update task doesn't make use of prepared statement batching
when writing job history, even though this is an obvious
and simple performance improvement.

This patch addresses these issues. Rather than blindly inserting a
new update task, a map is used to keep track of requests that have
already been queued for a DB update. Subsequent save attempts will
not create another update task. Thus only a single update task
at a time will work on a request, and it has the added benefit that
during periods of high load, several save operations can be
collapsed.

The update task itself is rewritten to use a single transaction,
to batch job history inserts and protocol table inserts, and to
release the request lock while executing DB statements.

Target: trunk
Request: 2.7
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Paul Millar
Patch: http://rb.dcache.org/r/5900/
(cherry picked from commit 8ff89b1b7c21c42dc18c5cfcc93907b32bac3307)