cvmfs/cvmfs — pull request #538 by reneme (2014-09-09 10:42:20+00:00)
Title: FIX: Remount With Open File Descriptors

This fixes (or at least mitigates) a potential race between `cvmfs_server publish` and anything touching the repository's mount point. Even though `cvmfs_server` checks for open file descriptors before running the synchronisation, it is possible that other processes fiddle (read-only) with the repository while the publish is running. See diabolic ticket [CVM-666](https://sft.its.cern.ch/jira/browse/CVM-666).

In this case the revision will be successfully published (and committed) but the remount of the new revision fails. This leaves the release manager mount points in an inconsistent state as happened recently to ATLAS.

There are three inconsistencies:

1. The AUFS mount point stays read-only, preventing a second try of `cvmfs_server publish` (see `health_check()` in `cvmfs_server`)
2. The new repository revision is successfully created and published but the release manager machine stays in the transaction
3. The base hash in `/var/spool/.../client.local` doesn't get updated

This fix does a second check for open file descriptors _after_ the synchronisation process has been finished. In this case, the remount is enforced and a warning message is printed. The enforcing happens just like the optional one when the open file descriptor was found before running the synchronisation. However, in the latter case, we do not have an option to keep the release manager consistency. Furthermore I moved the base-hash update further up in the procedure to be written right after the successful commit.

Additionally this contains a test case, that simulates the possible file descriptor situations:

* working directory on repository before publish
* open file descriptor on repository before publish with user option --> abort
* open file descriptor on repository before publish with user option --> continue
* open file descriptor race during publishing