pool: Fix cleaning of stale locations

dCache allows stale file locations. These are automatically cleaned when the
file is attempted to be accessed. The current implementation has a race if the
asynchronous notification to pnfs manager to remove the location is interleaved
with another thread creating that same file on the pool: In this case the
registration of the new location may succeed before the notification to clean
the location, thus leaving an orphaned file on the pool. The window for this is
very small and I therefore to not request the fix to be backported.

The patch fixes the problem by creating a temporary meta data entry for the
stale file. The entry is then immediately put into a REMOVED state, thus
causing the stale information to be cleaned. The temporary entry prevents other
threads from creating the file on the pool, thus preventing the race.

The fix comes at a cost of creating the meta data entry, but stale locations
should be sufficiently rare that the overhead is insignificant.

The patch fixes another problem affecting volatile pools: The notification
didn't include the flag to remove the name space entry once the last location
was removed.

The patch leaves another race open: The notification to clean the location sent
by the code removing the meta data entry is also asynchronous. Although the
code holds a lock on the meta data entry, the lock is eventually removed when
the entry is deleted. The pool could subsequently create a new entry or the
same file and send a registration message to the name space. If that message
were to be received before the notification to clean the previous location, we
would again have an orphaned file. For this race to be possible, messages
should be able to overtake each other. Without retries, messages between a pair
of pools should be ordered, which should make this race impossible. Fixes this
potential race would require changing the asynchronous notification to a
blocking request, and this would seriously affect sweeper performance and thus
uploads to full pools.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7502/