Package: looptime
Fast-forward asyncio event loop time (in tests)What?Fake the flow of time in asyncio event loops.
The effects of time removal can be seen from both sides:From theevent loop's (i.e. your tests') point of view,all external activities, such as synchronous executor calls (thread pools)
and i/o with sockets, servers, files, happen in zero amount of the loop time —
even if it takes some real time.
This hides the code overhead and network latencies from the time measurements,
making the loop time sharply and predictably advancing in configured steps.From theobserver's (i.e. your) point of view,all activities of the event loop, such as sleeps, events/conditions waits,
timeouts, "later" callbacks, happen in near-zero amount of the real time
(above the usual code overhead).
This speeds up the execution of tests without breaking the tests' time-based
design, even if they are designed to run in seconds or minutes.For the latter case, there are a few exceptions when the event loop's activities
are synced with the true-time external activities, such as thread pools or i/o,
so that they spend the real time above the usual code overhead (if configured).The library was originally developed forKopf,
a framework forKubernetes Operators in Python,
which actively uses asyncio tests in pytest (≈7000 unit-tests in ≈2 minutes).
You can see how this library changes and simplifies the tests inKopf's PR #881.Why?Withoutlooptime, the event loops usetime.monotonic()for the time,
which also captures the code overhead and the network latencies, adding little
random fluctuations to the time measurements (approx. 0.01-0.001 seconds).Withoutlooptime, the event loops spend the real wall-clock time
when there is no i/o happening but some callbacks are scheduled for later.
In controlled environments like unit tests and fixtures, this time is wasted.Also, because I can! (It was a little over-engineering exercise for fun.)ProblemIt is difficult to test complex asynchronous coroutines with the established
unit-testing practices since there are typically two execution flows happening
at the same time:One is for the coroutine-under-test which moves between states
in the background.Another one is for the test itself, which controls the flow
of that coroutine-under-test: it sets events, injects data, etc.In textbook cases with simple coroutines that are more like regular functions,
it is possible to design a test so that it runs straight to the end in one hop
— with all the preconditions set and data prepared in advance in the test setup.However, in real-world cases, the tests often must verify that
the coroutine stops at some point, waits for a condition for some limited time,
and then passes or fails.The problem is often "solved" by mocking the low-level coroutines of sleep/wait
that we expect the coroutine-under-test to call. But this violates the main
principle of good unit-tests: test the promise, not the implementation.
Mocking and checking the low-level coroutines is based on the assumptions
of how the coroutine is implemented internally, which can change over time.
Good tests do not change on refactoring if the protocol remains the same.Another (straightforward) approach is to not mock the low-level routines, but
to spend the real-world time, just in short bursts as hard-coded in the test.
Not only it makes the whole test-suite slower, it also brings the execution
time close to the values where the code overhead affects the timing
and makes it difficult to assert on the coroutine's pure time.SolutionSimilar to the mentioned approaches, to address this issue,looptimetakes care of mocking the event loop and removes this hassle from the tests.However, unlike the tests,looptimedoes not mock the typically used
low-level coroutines (e.g. sleep), primitives (e.g. events/conditions),
or library calls (e.g. requests getting/posting, sockets reading/writing, etc).looptimegoes deeper and mocks the very foundation of it all — the time itself.
Then, it controllably moves the time forward in sharp steps when the event loop
requests the actual true-time sleep from the underlying selectors (i/o sockets).ExamplesHere, we assume that the async tests are supported. For example,
usepytest-asyncio:pipinstallpytest-asyncio
pipinstalllooptimeNothing is needed to make async tests run with the fake time, it just works:importasyncioimportpytest@pytest.mark.asyncioasyncdeftest_me():awaitasyncio.sleep(100)assertasyncio.get_running_loop().time()==100pytest--looptimeThe test will be executed in approximately0.01 seconds,
while the event loop believes it is 100 seconds old.If the command line or ini-file options for all tests is not desirable,
individual tests can be marked for fast time forwarding explicitly:importasyncioimportpytest@pytest.mark.asyncio@pytest.mark.looptimeasyncdeftest_me():awaitasyncio.sleep(100)assertasyncio.get_running_loop().time()==100pytestUnder the hood, the library solves some nuanced situations with time in tests.
See "Nuances" below for more complicated (and nuanced) examples.Markers@pytest.mark.looptimeconfigures the test's options if and when it is
executed with the timeline replaced to fast-forwarding time.
In normal mode with no configs/CLI options specified,
it marks the test to be executed with the time replaced.@pytest.mark.looptime(False)(with the positional argument)
excludes the test from the time fast-forwarding under any circumstances.
The test will be executed with the loop time aligned with the real-world time.
Use it only for the tests that are designed to be true-time-based.Note that markers can be applied not only to individual tests,
but also to whole test suites (classes, modules, packages):importasyncioimportpytestpytestmark=[pytest.mark.asyncio,pytest.mark.looptime(end=60),]asyncdeftest_me():awaitasyncio.sleep(100)The markers can also be artificially injected by plugins/hooks if needed:importasyncioimportpytest@pytest.hookimpl(hookwrapper=True)defpytest_pycollect_makeitem(collector,name,obj):ifcollector.funcnamefilter(name)andasyncio.iscoroutinefunction(obj):pytest.mark.asyncio(obj)pytest.mark.looptime(end=60)(obj)yieldAll in all, thelooptimeplugin uses the most specific (the "closest") value
for each setting separately (i.e. not the closest marker as a whole).Options--looptimeenables time fast-forwarding for all tests that are not explicitly
marked as using the fake loop time —including those not marked at all—
as if all tests were implicitly marked.--no-looptimeruns all tests —both marked and unmarked— with the real time.
This flag effectively disables the plugin.SettingsThe marker accepts several settings for the test. The closest to the test
function applies. This lets you define the test-suite defaults
and override them on the directory, module, class, function, or test level:importasyncioimportpytestpytestmark=pytest.mark.looptime(end=10,idle_timeout=1)@pytest.mark.asyncio@pytest.mark.looptime(end=101)asyncdeftest_me():awaitasyncio.sleep(100)assertasyncio.get_running_loop().time()==100The time zerostart(floatorNone, or a no-argument callable that returns the same)
is the initial time of the event loop.If it is callable, it is invoked once per event loop to get the value:
e.g.start=time.monotonicto align with the true time,
orstart=lambda: random.random() * 100to add some unpredictability.Noneis treated the same as0.0.The default is0.0.The end of timeend(floatorNone, or a no-argument callable that returns the same)
is the final time in the event loop (the internal fake time).
If it is reached, all tasks get terminated and the test is supposed to fail.
The injected exception isLoopTimeoutError,
a subclass ofasyncio.TimeoutError.All test-/fixture-finalizing routines will have their fair chance to execute
as long as they do not move the loop time forward, i.e. they take zero time:
e.g. withasyncio.sleep(0), simpleawaitstatements, etc.If set toNone, there is no end of time, and the event loop runs
as long as needed. Note:0means ending the time immediately on start.If it is a callable, it is called once per event loop to get the value:
e.g.end=lambda: time.monotonic() + 10.The end of time is not the same as timeouts — see the nuances below
on differences withasync-timeout.NuancesPreliminary executionConsider this test:importasyncioimportasync_timeoutimportpytest@pytest.mark.asyncio@pytest.mark.looptimeasyncdeftest_me():asyncwithasync_timeout.timeout(9):awaitasyncio.sleep(1)Normally, it should not fail. However, with fake time (without workarounds)
the following scenario is possible:async_timeoutlibrary sets its delayed timer at 9 seconds since now.the event loop notices that there is only one timer at T0+9s.the event loop fast-forwards time to be9.since there are no other handles/timers, that timer is executed.async_timeoutfails the test withasyncio.TimeoutErrorThesleep()never gets any chance to be scheduled or executed.To solve this,looptimeperforms several dummy zero-time no-op cycles
before actually moving the time forward. This gives other coroutines,
tasks, and handles their fair chance to be entered, spawned, scheduled.
This is why the example works as intended.Thenoop_cycles(int) setting is how many cycles the event loop makes.
The default is42. Why 42? Well, …Slow executorsConsider this test:importasyncioimportasync_timeoutimportcontextlibimportpytestimportthreadingdefsync_fn(event:threading.Event):event.set()@pytest.mark.asyncio@pytest.mark.looptimeasyncdeftest_me(event_loop):sync_event=threading.Event()withcontextlib.suppress(asyncio.TimeoutError):asyncwithasync_timeout.timeout(9):awaitevent_loop.run_in_executor(None,sync_fn,sync_event)assertsync_event.is_set()With the true time, this test will finish in a fraction of a second.
However, with the fake time (with no workarounds), the following happens:A new synchronous event is created, it is unset by default.A synchronous task is submitted to a thread pool executor.The thread pool starts spawning a new thread and passing the task there.An asynchronous awaitable (future) is returned, which is chained
with its synchronous counterpart.looptimeperforms its no-op cycles, letting all coroutines to start,
but it does this in near-zero true-time.The event loop forwards its time to 9 seconds and raises a timeout error.The test suppresses the timeout, checks the assertion, and fails:
the sync event is still unset.A fraction of a second (e.g.0.001second) later, the thread starts,
calls the function and sets the sync event, but it is too late.Compared to the fake fast-forwarding time, even such fast things as threads
are too slow to start. Unfortunately,looptimeand the event loop can
neither control what is happening outside of the event loop nor predict
how long it will take.To work around this,looptimeremembers all calls to executors and then
keeps track of the futures they returned. Instead of fast-forwarding the time
by 9 seconds all at once,looptimefast-forwards the loop's fake time
in small steps and also does the true-time sleep for that step.
So, the fake time and real time move along while waiting for executors.Luckily for this case, in 1 or 2 such steps, the executor's thread will
do its job, the event will be set, so as the synchronous & asynchronous
futures of the executor. The latter one (the async future) will also
let theawaitmove on.Theidle_step(floatorNone) setting is the duration of a single
time step when fast-forwarding the time if there are executors used —
i.e. if some synchronous tasks are running in the thread pools.Note that the steps are both true-time and fake-time: they spend the same
amount of the observer's true time as they increment the loop's fake time.A negative side effect: the thread spawning can be potentially much faster,
e.g. finish in in 0.001 second; but it will be rounded to be the round number
of steps with no fractions: e.g. 0.01 or 0.02 seconds in this example.A trade-off: the smaller step will get the results faster,
but will spend more CPU power on resultless cycles.I/O idleConsider this test:importaiohttpimportpytest@pytest.mark.asyncio@pytest.mark.looptimeasyncdeftest_me():asyncwithaiohttp.ClientSession(timeout=None)assession:awaitsession.get('http://some-unresponsive-web-site.com')How long should it take if there are no implicit timeouts deep in the code?
With no workarounds, the test will hang forever waiting for the i/o to happen.
This mostly happens when the only thing left in the event loop is the i/o,
all internal scheduled callbacks are gone.looptimecan artificially limit the lifetime of the event loop.
This can be done as a default setting for the whole test suite, for example.Theidle_timeout(floatorNone) setting is the true-time limit
of the i/o wait in the absence of scheduled handles/timers/timeouts.
(This i/o includes the dummy i/o used byloop.call_soon_threadsafe().)Nonemeans there is no timeout waiting for the i/o, i.e. it waits forever.
The default is1.0seconds.If nothing happens within this time, the event loop assumes that nothing
will happen ever, so it is a good idea to cease its existence: it injectsIdleTimeoutError(a subclass ofasyncio.TimeoutError) into all tasks.This is similar to how the end-of-time behaves, except that it is measured
in the true-time timeline, while the end-of-time is the fake-time timeline.
Besides, once an i/o happens, the idle timeout is reset, while the end-of-time
still can be reached.Theidle_step(floatorNone) setting synchronises the flow
of the fake-time with the flow of the true-time while waiting for the i/o
or synchronous futures, i.e. when nothing happens in the event loop itself.
It sets the single step increment of both timelines.If the step is not set or set toNone, the loop time does not move regardless
of how long the i/o or synchronous futures take in the true time
(with or without the timeout).If theidle_stepis set, but theidle_timeoutisNone,
then the fake time flows naturally in sync with the true time infinitely.The default isNone.Timeouts vs. the end-of-timeThe end of time might look like a global timeout, but it is not the same,
and it is better to use other methods for restricting the execution time:
e.g.async-timeoutor nativeasyncio.wait_for(…, timeout=…).First, the mentioned approaches can be applied to arbitrary code blocks,
even multiple times independently,
whilelooptime(end=N)applies to the lifecycle of the whole event loop,
which is usually the duration of the whole test and monotonically increases.Second,looptime(end=N)syncs the loop time with the real time for N seconds,
i.e. it does not instantly fast-forward the loop time when the loops
attempts to make an "infinite sleep" (technically,selector.select(None)).async_timeout.timeout()andasyncio.wait_for()set a delayed callback,
so the time fast-forwards to it on the first possible occasion.Third, once the end-of-time is reached in the event loop, all further attempts
to run async coroutines will fail (except those taking zero loop time).
If the async timeout is reached, further code can proceed normally.importasyncioimportpytest@pytest.mark.asyncio@pytest.mark.looptime(end=10)asyncdeftest_the_end_of_time(chronometer,looptime):withchronometer:withpytest.raises(asyncio.TimeoutError):awaitasyncio.Event().wait()assertlooptime==10assertchronometer>=10@pytest.mark.asyncio@pytest.mark.looptimeasyncdeftest_async_timeout(chronometer,looptime):withchronometer:withpytest.raises(asyncio.TimeoutError):awaitasyncio.wait_for(asyncio.Event().wait(),timeout=10)assertlooptime==10assertchronometer<0.1Time resolutionPython (so as many other languages) has issues with calculating the floats:>>> 0.2-0.05
0.15000000000000002
>>> 0.2-0.19
0.010000000000000009
>>> 0.2+0.21
0.41000000000000003
>>> 100_000 * 0.000_001
0.09999999999999999This can break the assertions on the time and durations. To work around
the issue,looptimeinternally performs all the time math in integers.
The time arguments are converted to the internal integer form
and back to the floating-point form when needed.Theresolution(float) setting is the minimum supported time step.
All time steps smaller than that are rounded to the nearest value.The default is 1 microsecond, i.e.0.000001(1e-6), which is good enough
for typical unit-tests while keeps the integers smaller than 32 bits
(1 second => 20 bits; 32 bits => 4294 seconds ≈1h11m).Normally, you should not worry about it or configure it.A side-note: in fact, the reciprocal (1/x) of the resolution is used.
For example, with the resolution0.001, the time1.0(float) becomes1000(int),0.1(float) becomes100(int),0.01(float) becomes10(int),0.001(float) becomes1(int);
everything smaller than0.001becomes0and probably misbehaves.ExtrasChronometersFor convenience, the library also provides a class and a fixture
to measure the duration of arbitrary code blocks in real-world time:looptime.Chronometer(a context manager class).chronometer(a pytest fixture).It can be used as a sync or async context manager:importasyncioimportpytest@pytest.mark.asyncio@pytest.mark.looptimeasyncdeftest_me(chronometer):withchronometer:awaitasyncio.sleep(1)awaitasyncio.sleep(1)assertchronometer.seconds<0.01# random code overheadUsually, the loop-time duration is not needed or can be retrieved viaasyncio.get_running_loop().time(). If needed, it can be measured using
the provided context manager class with the event loop's clock:importasyncioimportlooptimeimportpytest@pytest.mark.asyncio@pytest.mark.looptime(start=100)asyncdeftest_me(chronometer,event_loop):withchronometer,looptime.Chronometer(event_loop.time)asloopometer:awaitasyncio.sleep(1)awaitasyncio.sleep(1)assertchronometer.seconds<0.01# random code overheadassertloopometer.seconds==2# precise timing, no code overheadassertevent_loop.time()==102Loop time assertionsThelooptimefixtureis syntax sugar for easy loop time assertions::importasyncioimportpytest@pytest.mark.asyncio@pytest.mark.looptime(start=100)asyncdeftest_me(looptime):awaitasyncio.sleep(1.23)assertlooptime==101.23Technically, it is a proxy object toasyncio.get_running_loop().time().
The proxy object supports the direct comparison with numbers (integers/floats),
so as some basic arithmetics (adding, subtracting, multiplication, etc).
However, it adjusts to the time precision of 1 nanosecond (1e-9): every digit
beyond that precision is ignored — so you can be not afraid of123.456/1.2suddenly becoming102.88000000000001and not equal to102.88(as long as the time proxy object is used and not converted to a native float).The proxy object can be used to create a new proxy that is bound to a specific
event loop (it works for loops both with fake- and real-world time)::importasynciofromlooptimeimportpatch_event_loopdeftest_me(looptime):new_loop=patch_event_loop(asyncio.new_event_loop(),start=100)new_loop.run_until_complete(asyncio.sleep(1.23))assertlooptime@new_loop==101.23Mind that it is not the same asChronographerfor the whole test.
The time proxy reflects the time of the loop, not the duration of the test:
the loop time can start at a non-zero point; even if it starts at zero,
the loop time also includes the time of all fixtures setups.Custom event loopsDo you use a custom event loop? No problem! Create a test-specific descendant
with the provided mixin — and it will work the same as the default event loop.importlooptimeimportpytestfromwhereverimportCustomEventLoopclassLooptimeCustomEventLoop(looptime.LoopTimeEventLoop,CustomEventLoop):pass@pytest.fixturedefevent_loop():returnLooptimeCustomEventLoop()Only selector-based event loops are supported: the event loop must rely onself._selector.select(timeout)to sleep fortimeouttrue-time seconds.
Everything that inherits fromasyncio.BaseEventLoopshould work.You can also patch almost any event loop class or event loop object
the same way aslooptimedoes that (via some dirty hackery):importasyncioimportlooptimeimportpytest@pytest.fixturedefevent_loop():loop=asyncio.new_event_loop()returnlooptime.patch_event_loop(loop)looptime.make_event_loop_class(cls)constructs a new class that inherits
from the referenced class and the specialised event loop class mentioned above.
The resulting classes are cached, so it can be safely called multiple times.looptime.patch_event_loop()replaces the event loop's class with the newly
constructed one. For those who care, it is an equivalent of the following hack
(some restrictions apply to the derived class):loop.__class__=looptime.make_event_loop_class(loop.__class__)