Repository: dask/distributed

Issue:
Need Window Timer
We need a timer for Windows that has the following properties:

1.  Fast to run (microsecond or less) - like `time.time` or `timeit.default_timer`
2.  Roughly globally consistent (to within minutes) - like `time.time` but not `timeit.default_timer`
3.  Microsecond precision - like `timeit.default_timer`, but not `time.time`

I wonder if there is a way to combine `time.time` and `timeit.default_timer` to achieve the desired result.  Or is there an established alternative?

cc @pitrou @jreback 


Discussion:
"Roughly globally consistent" - you mean accross different machines?
On Windows:
* `time.time()` is the absolute time but has poor resolution
* `time.process_time()` is wall clock time with a fine resolution but an unspecified reference point

By combining the two we should be able to get "absolute" times with decent resolution.
About performance:
```
>>> %timeit time.time()
The slowest run took 132.95 times longer than the fastest. This could mean that
an intermediate result is being cached.
10000000 loops, best of 3: 81.9 ns per loop
>>> %timeit time.perf_counter()
The slowest run took 4.79 times longer than the fastest. This could mean that an
 intermediate result is being cached.
100000 loops, best of 3: 3.44 µs per loop
```

This makes `perf_counter()` a bit slow, but not that much in the context of Python bytecode execution.
In the Dask scheduler we maintain a known lag-time to each worker that we use to correct all incoming times.  Usually this lag-time is on the order of seconds and we trust that the value never rolls over.  

We probably could use `default_timer` and be fine, especially if the workers are ephemeral, but I'd trust things more if everyone used a baseline that was roughly consistent.
Here is a proposal: https://gist.github.com/pitrou/faf437bf417c77c3cd96b1071737a90b

```
>>> [(time.time(), synctime.synctime()) for i in range(20)]

[(1480609651.5324142, 1480609651.5332496),
 (1480609651.5324142, 1480609651.5332682),
 (1480609651.5324142, 1480609651.5332782),
 (1480609651.5324142, 1480609651.5332887),
 (1480609651.5324142, 1480609651.533298),
 (1480609651.5324142, 1480609651.5333087),
 (1480609651.5324142, 1480609651.5333183),
 (1480609651.5324142, 1480609651.5333276),
 (1480609651.5324142, 1480609651.5333366),
 (1480609651.5324142, 1480609651.5333467),
 (1480609651.5324142, 1480609651.5333564),
 (1480609651.5324142, 1480609651.5333657),
 (1480609651.5324142, 1480609651.533375),
 (1480609651.5324142, 1480609651.5333846),
 (1480609651.5324142, 1480609651.5333939),
 (1480609651.5324142, 1480609651.5334034),
 (1480609651.5324142, 1480609651.533413),
 (1480609651.5324142, 1480609651.5334232),
 (1480609651.5324142, 1480609651.533433),
 (1480609651.5324142, 1480609651.5334423)]
>>> %timeit synctime.synctime()
The slowest run took 5.39 times longer than the fastest. This could mean that an
 intermediate result is being cached.
100000 loops, best of 3: 4.46 µs per loop
```

I agree using a reliable baseline is better. Also, modern systems are supposed to sync their internal clocks with NTP.
Do you want me to make a PR out of that?
(note this scheme doesn't guarantee monotonicity, although it should be statistically monotonic :-))
(even time.time() isn't guaranteed to be monotonic, though...)
Is drift something to worry about?  Is one `perf_counter` not fairly precisely a second?
I don't know how precisely. If a daemon runs for multiple days or weeks, then perhaps it'll start drifting. As far as I know, `perf_counter` can be based on the CPU's performance counters, which are entirely independent from the system's absolute time.
(also, the performance counters won't be adjusted in case of e.g. DST switch)
A fairly comprehensive PEP was written about the general problem, by the way: https://www.python.org/dev/peps/pep-0418/#windows-queryperformancecounter
Should the 360 number be much larger then?
Anyway, yes, I think a PR would be good
Well, resync() should be fairly cheap (around 65µs here), so I think it's better to be conservative...
OK

Resolving patch:
diff --git a/distributed/metrics.py b/distributed/metrics.py
old mode 100644
new mode 100755
--- a/distributed/metrics.py
+++ b/distributed/metrics.py
@@ -2,6 +2,8 @@
 
 import collections
 from functools import wraps
+import sys
+import time as timemod
 
 
 _empty_namedtuple = collections.namedtuple("_empty_namedtuple", ())
@@ -29,3 +31,54 @@ def wrapper():
 
 net_io_counters = _psutil_caller("net_io_counters")
 
+
+class _WindowsTime(object):
+    """
+    Combine time.time() and time.perf_counter() to get an absolute clock
+    with fine resolution.
+    """
+
+    # Resync every N seconds, to avoid drifting
+    RESYNC_EVERY = 600
+
+    def __init__(self):
+        self.delta = None
+        self.last_resync = float("-inf")
+
+    if sys.version_info >= (3,):
+        perf_counter = timemod.perf_counter
+    else:
+        perf_counter = timemod.clock
+
+    def time(self):
+        delta = self.delta
+        cur = self.perf_counter()
+        if cur - self.last_resync >= self.RESYNC_EVERY:
+            delta = self.resync()
+            self.last_resync = cur
+        return delta + cur
+
+    def resync(self):
+        _time = timemod.time
+        _perf_counter = self.perf_counter
+        min_samples = 5
+        while True:
+            times = [(_time(), _perf_counter()) for i in range(min_samples * 2)]
+            abs_times = collections.Counter(t[0] for t in times)
+            first, nfirst = abs_times.most_common()[0]
+            if nfirst < min_samples:
+                # System too noisy? Start again
+                continue
+            else:
+                perf_times = [t[1] for t in times if t[0] == first][:-1]
+                assert len(perf_times) >= min_samples - 1, perf_times
+                self.delta = first - sum(perf_times) / len(perf_times)
+                return self.delta
+
+
+# A high-resolution wall clock timer measuring the seconds since Unix epoch
+if sys.platform.startswith('win'):
+    time = _WindowsTime().time
+else:
+    # Under modern Unices, time.time() should be good enough
+    time = timemod.time
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -10,7 +10,6 @@
 import random
 import tempfile
 from threading import current_thread, Lock, local
-from time import time
 from timeit import default_timer
 import shutil
 import sys
@@ -35,6 +34,7 @@
 from .compatibility import reload, unicode
 from .core import (read, write, connect, close, send_recv, error_message,
                    rpc, Server, pingpong, coerce_to_address, RPCClosed)
+from .metrics import time
 from .protocol.pickle import dumps, loads
 from .sizeof import sizeof
 from .threadpoolexecutor import ThreadPoolExecutor


Test patch:
diff --git a/distributed/tests/test_metrics.py b/distributed/tests/test_metrics.py
new file mode 100644
--- /dev/null
+++ b/distributed/tests/test_metrics.py
@@ -0,0 +1,21 @@
+from __future__ import print_function, division, absolute_import
+
+import time
+
+import pytest
+
+from distributed import metrics
+
+
+def test_wall_clock():
+    for i in range(3):
+        time.sleep(0.01)
+        t = time.time()
+        samples = [metrics.time() for j in range(50)]
+        # Resolution
+        deltas = [samples[j+1] - samples[j] for j in range(len(samples) - 1)]
+        assert min(deltas) >= 0.0, deltas
+        assert max(deltas) <= 1.0, deltas
+        assert any(lambda d: 0.0 < d < 0.0001 for d in deltas), deltas
+        # Close to time.time()
+        assert t - 0.5 < samples[0] < t + 0.5