webadmin: make srmSpaceManagerEnabled aware; enable real backoff on coll...

...ectors

module: webadmin

In order to populate the webadmin pages, data collector daemons are run which contact various services such as the PoolManager or the SpaceManager.  While it would be uncommon to run dCache with most of these services, it is not uncommon for the SpaceManager not to be there.  In this case, we would like to avoid running this collector; and in general, we would also like to backoff from pinging services which may be temporarily offline.

Some time ago, a backoff algorithm factory was added to use precisely for this reason.  However, the original constant recurrent message send/wait was for the moment maintained.

This patch does the following:

   - it passes through knowledge of the srmSpaceManagerEnabled property.  It is assumed here that the local dcache.conf file will reflect the actual setting, whereever the SpaceManager may be configured to run;

   - it slightly restructures several of the collectors so that backoff will occur at the correct place in the looping logic;
   - it creates and adds an exponential backoff algorithm to be used by the collectors most susceptible to not finding their single endpoints (PoolMonitor, RestoreHandler, SpaceToken).

Target: master
Committed: master@92687f8010e0dfa60412aaef316e5aff80411769
Patch: http://rb.dcache.org/r/5571/
Require-notes: no
Require-book: no
Request: 2.6
Acked-by: Tigran

Redeployed.

1.  With default, the SpaceToken collector does not run (srmSpaceManagerEnabled=no)
2.  Setting srmSpaceManagerEnabled=yes, but with no actual space manager running, observed the correct backoff behavior
3.  Adding space manager and starting the domain allowed the collector to connect on its subsequent iteration.