test_latency.py: Try do to the receive buffering right.