dCache/dcache — pull request #1124 by gbehrmann (2014-11-27 09:34:42+00:00)
Title: spacemanager: Fix various error recovery scenarios

The patch fixes several related issues in space manager:

- The code used to delete a file reservation when the mover reported
a failure. However whether such a file is actually deleted depends
on what the door does. If the door deletes it, the space manager
will be notified and the file can be deleted at that point.

The special case that exists is if the mover failure was caused by the
pool being unable to register the location in the name space. When
such a file is deleted it would only get a name space delete notification,
and no pool remove file notification. To account for this case, the
pool leaves a file for which the mover reported an error in the
transferring state. If the name space entry is deleted, such an entry
is removed. If the file is not deleted, the other fixes of this
patch will address it eventually.

- The code used to assume that a file that was in transferring
would either get a DoorTransferFinished notification or be deleted.
There is however no guarantee that this happens for non-srm uploads,
such as when the door doesn't delete on failed upload. The client
may even think the transfer is successfull if no control channel is
used (eg http and xrootd). Such a file is perfectly readable, but
stayed in transferring in space manager.

This patch addresses this problem by extending the background check
of files that have stayed in transferring for more than 24 hours:
rather than just deleting reservations for deleted files, the code
now also detects flushed and completed files and updates the reservation
accordingly.

- The patch changes how we react to notifications that a name space entry
is deleted: Before, we deleted the reservation if the file was marked
flushed. For other states we assumed we would eventually get a pool
remove file notification: However if the pool failed to register the
location, we will never get such a notification. This scenario is
quite likely if the name space entry is deleted during upload.

The patch addresses this problem by deleting a transferring reservation on
the name space delete notification. The downside is that we have some
space allocated for the file on the pool that isn't explicitly linked
to a reservation in space manager, but since the name space entry got
deleted, the file on the pool will soon get deleted too.

- The patch also refines how we react to notifications for flushed files: if the
reservation was in transferring, it is now considered complete. Before the
flush notification was simply ignored. This could both happen for lost
DoorTransferFinished notification, of if the flush notification managed to
overtake the DoorTransferFinished notification (unlikely).

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev
Patch: https://rb.dcache.org/r/7519/
(cherry picked from commit c59f61eb89d04948f7e8172b15cbd8288778533c)