spacemanager: Fix various error recovery scenarios

The patch fixes several related issues in space manager:

- The code used to delete a file reservation when the mover reported
  a failure. However whether such a file is actually deleted depends
  on what the door does. If the door deletes it, the space manager
  will be notified and the file can be deleted at that point.

  The special case that exists is if the mover failure was caused by the
  pool being unable to register the location in the name space. When
  such a file is deleted it would only get a name space delete notification,
  and no pool remove file notification. To account for this case, the
  pool leaves a file for which the mover reported an error in the
  transferring state. If the name space entry is deleted, such an entry
  is removed. If the file is not deleted, the other fixes of this
  patch will address it eventually.

- The code used to assume that a file that was in transferring
  would either get a DoorTransferFinished notification or be deleted.
  There is however no guarantee that this happens for non-srm uploads,
  such as when the door doesn't delete on failed upload. The client
  may even think the transfer is successfull if no control channel is
  used (eg http and xrootd). Such a file is perfectly readable, but
  stayed in transferring in space manager.

  This patch addresses this problem by extending the background check
  of files that have stayed in transferring for more than 24 hours:
  rather than just deleting reservations for deleted files, the code
  now also detects flushed and completed files and updates the reservation
  accordingly.

- The patch changes how we react to notifications that a name space entry
  is deleted: Before, we deleted the reservation if the file was marked
  flushed. For other states we assumed we would eventually get a pool
  remove file notification: However if the pool failed to register the
  location, we will never get such a notification. This scenario is
  quite likely if the name space entry is deleted during upload.

  The patch addresses this problem by deleting a transferring reservation on
  the name space delete notification. The downside is that we have some
  space allocated for the file on the pool that isn't explicitly linked
  to a reservation in space manager, but since the name space entry got
  deleted, the file on the pool will soon get deleted too.

- The patch also refines how we react to notifications for flushed files: if the
  reservation was in transferring, it is now considered complete.  Before the
  flush notification was simply ignored. This could both happen for lost
  DoorTransferFinished notification, of if the flush notification managed to
  overtake the DoorTransferFinished notification (unlikely).

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: https://rb.dcache.org/r/7519/
(cherry picked from commit c59f61eb89d04948f7e8172b15cbd8288778533c)
info: Reduce delay between messages

The info service takes a long time to populate its state after a restart, in
particular on production scale systems.

This patch reduces the delay between queries. The overall frequency is not
increased, but the delay between two queries is reduced to 100 ms (from 2, 5,
or 10 seconds, depending on the query). As these queries are relatively
light-weight, 100 ms should be plenty to keep the message passing overhead
down.

I request backport to 2.10 as we always have issues with our Nagios probes
after a restart.

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7535/
(cherry picked from commit 9317083dcb67512cb72636b4802e152db55cd7fa)
pool: Fix ISE when flushing deleted file

Prior to flushing a file, the pool checks if the name space entry still exists. If
not, it deletes the replica rather than flushing it to tape. Release 2.10.7 introduced
a regression that causes this code path to throw an ISE:

11:41:01 [pool-28-thread-1] [1f45dec7-f415-4dea-80cb-57c825b51486] Flush of 000084C99C4AC70E4CC38279851A856FABB8 failed with: CacheException(rc=10001;msg=No such file or directory: 000084C99C4AC70E4CC38279851A856FABB8).
11:41:01 [pool-12-thread-2] [1f45dec7-f415-4dea-80cb-57c825b51486] remove entry for: 000084C99C4AC70E4CC38279851A856FABB8
11:41:02 [pool-10-thread-1] [door:GFTP-Gerds-MacBook-Pro-5-AAUHz0KyLFg GFTP-Gerds-MacBook-Pro-5-AAUHz0KyLFg PoolAcceptFile 000084C99C4AC70E4CC38279851A856FABB8] Unexpected failure during state change notificationjava.lang.IllegalStateException: Handle is closed
    at org.dcache.pool.repository.v5.ReadHandleImpl.close(ReadHandleImpl.java:48) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.nearline.NearlineStorageHandler$FlushRequestImpl.failed(NearlineStorageHandler.java:628) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.nearline.AbstractBlockingNearlineStorage$Task.cancel(AbstractBlockingNearlineStorage.java:200) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.nearline.AbstractBlockingNearlineStorage.cancel(AbstractBlockingNearlineStorage.java:71) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.nearline.NearlineStorageHandler$AbstractRequest.cancel(NearlineStorageHandler.java:365) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.nearline.NearlineStorageHandler$AbstractRequestContainer.cancel(NearlineStorageHandler.java:446) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.nearline.NearlineStorageHandler.stateChanged(NearlineStorageHandler.java:279) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.repository.v5.StateChangeListeners$1.run(StateChangeListeners.java:68) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]

11:41:02 [pool-12-thread-2] [1f45dec7-f415-4dea-80cb-57c825b51486] File not found in name space; removed 000084C99C4AC70E4CC38279851A856FABB8.

This is caused by the pre-flush check deleting the replica followed by failing the flush
request. Deleting the replica subsequently causes a notification to be sent to the HSM
subsystem, which causes the same flush request to be cancelled. The script driver reports
a second failure, thus causing the replica handler to be closed twice - hence the ISE.

The patch fixes the regression.

The log message is also cleaned up to better state what happened:

12:07:39 [pool-28-thread-1] [84967682-8766-4e66-80c3-3199f047b6b2] Flush of 00005466F5DC5AEA4263B3FA8E6EECCCF045 failed with: CacheException(rc=10001;msg=File not found in name space during pre-flush check.).
12:07:39 [pool-12-thread-2] [84967682-8766-4e66-80c3-3199f047b6b2] remove entry for: 00005466F5DC5AEA4263B3FA8E6EECCCF045

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7506/
(cherry picked from commit 61dfae7938bbb171d8ac777fb182ef512ff77f5f)
pool: reorder ip addresses returned to NFS client

according to NFS spec pool client can select a best matching IP
address from the list available IP addresses of the pool.

Nevertheless, current client implementation in RHEL-6 takes first
one from the list and, some times, makes a wrong choice for mutihomed
pools.

This change will put 'expected correct' interface as a first element
of the returned list.

Ticket: #8404
Acked-by: Gerd Behrmann
Target: master, 2.11, 2.10, 2.9, 2.8, 2.7, 2.6
Require-book: no
Require-notes: no
(cherry picked from commit 7c43c2ad6a6d6209625ee78d6745a5f8c531d7a3)
Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
info: Reduce safety limit to 50 ms

After reducing the DGA query separation to 100 ms, the DGA safety kick in and
limits the DGA frequency. This patch changes the safety limit to 50 ms (20 Hz).

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7540/
(cherry picked from commit 8c295eeec9da72895b1994d44eebff7b522edee7)
Upgrade to Milton 2.6

Appears to be working. Solves the problem where proxied vector reads cause the entire
file to be written to local disk. Instead only the actual data that is to be returned
to the client is buffered. If this is more than 100 KB, that data is buffered on local
disk. The file is deleted after successful return of the data.

There are still problems though: In case of failures reading the data from the pool,
the temporary file is not deleted. The entire file is still downloaded from the pool
to the door (even when only the requested data is buffered), and the multi-range writer
in Milton is extremely inefficient as it processes one byte at a time (it fails to
override the multi-byte write methods).

The new version supports using cookies to remeber when a client has already logged in.
I disabled this feature for now.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7530/
(cherry picked from commit b0f88c5080926e867ca644087a4e75e7c077bf00)

Conflicts:
	pom.xml
ftp: Relax requirements for EPSV and EPRT

Since using EPSV without delayed passive forces the use of a data channel
proxy, we have rejected use of RFC 2428 commands over IPv4. This requirement
can however be relaxed: EPRT can be used over IPv4 without proxying, and
EPSV can be used over IPv4 without proxying when delayed passive is enabled.

The patch has the potential to extend compatibility with clients and I hence
request it to be backported to 2.10.

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7537/
(cherry picked from commit 1d4fb5882fcaa898fe740ea983e9676a51d1b992)
srm,webdav,httpd: Log SSL handshake failures

Will log SSL failures, such as when clients try to use disabled features. Eg:

28 nov. 2014 10:21:53 (SRM-Gerds-MacBook-Pro-5) [] SSL handshake with localhost/127.0.0.1:62713 failed: SSLv2Hello is disabled

Target: trunk
Request: 2.11
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7539/
(cherry picked from commit 3f52dce613e1f76efe119da11a8fab18e711b2d8)
poolmanager: Fix full pool detection for WASS

When a file cannot fit within the free plus removable space without
crossing the pool gap, a pool is considered full. WASS erroneously
used the decayed removable space in this check. This meant that
if all pools had very fresh files, they could all be considered
full even though there are plenty of removable files. This patch
changes the check to consider all removable space of a pool. The
decayed removable space is only used when weighting the random
selection.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8390
Require-notes: yes
Require-book: no
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Patch: https://rb.dcache.org/r/7531/
(cherry picked from commit 42e31b5f935af4f3220aa29f2e51e8ea2c0bd146)
Merge pull request #1143 from gbehrmann/fix/2.11/rb7539

srm,webdav,httpd: Log SSL handshake failures
Merge pull request #1144 from gbehrmann/fix/2.11/rb7531

poolmanager: Fix full pool detection for WASS
Merge pull request #1135 from gbehrmann/fix/2.11/rb7530

Upgrade to Milton 2.6
Merge pull request #1134 from kofemann/nfs-no-multipath-2.11

pool: reorder ip addresses returned to NFS client
Merge pull request #1125 from gbehrmann/fix/2.11/rb7535

info: Reduce delay between messages
Merge pull request #1127 from gbehrmann/fix/2.11/rb7506

pool: Fix ISE when flushing deleted file
Merge pull request #1122 from gbehrmann/fix/2.11/rb7519

spacemanager: Fix various error recovery scenarios
nfs: set file size in namespace on stable_how is FILE_SYNC4

In some workloads, nfs client may request file size before mover
is slosed and file size is set. Nevertheless, prior such requests client
must commit written data by sending an explicit COMMIT operation
or WRITE with FILE_SYNC flag.

This change covers both cases. To match pnfsid's to mover we keep
a dedicated mapping with an assumption, that only single mover can
perform write operation.

2.11 is requested as we see the need for a sync in production system at desy.

Acked-by: Gerd Behrmann
Target: master, 2.11
Require-book: no
Require-notes: yes
(cherry picked from commit 3dca7ee2823a8157214ca351aa2013822c909c85)
Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
nfs: return correct verifier on WRITE and COMMIT

Acked-by: Paul Millar
Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
(cherry picked from commit 952316ff36f15b7f4dec962ad279994dcd97bacd)
Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
ftp: Relax requirements for EPSV and EPRT

Fix unit tests after 1d4fb58.

(cherry picked from commit af625e3cf88fe40a1d009dc46c1c6018047511bd)
Merge pull request #1141 from gbehrmann/fix/2.11/rb7537

ftp: Relax requirements for EPSV and EPRT