Repository: python/mypy

Issue:
Tests not parallelized efficienctly by pytest
When I run `pytest mypy`, towards the end of a test run only 1 or 2 python processes are running and test progress is very slow -- each process is using much less than 100% of a CPU. This happens even though I requested 8 parallel processes. Initially I see 8 processes doing work, but something causes a slowdown towards the end of a test run. 

This doesn't look like a recent regression, but I think that things have gotten worse in the last few months or so.

Some data (Linux, Intel 4-core/8-thread CPU):
* When I check out e3786018ddcdac065a89b2db9889fb04165ee71e from May 2018, tests finish in 66s (6024 tests).
* When I check out 1606f0f9174dd168fa4848b5dafdc27174c67e9e from Oct 2018, tests finish in 133s (6671 tests).

This seems to happen both on Linux and macOS.

Updating to the latest pytest doesn't help.

It would be great if somebody could try to reproduce the issue.


Discussion:
Yes, I have seen this too. The problem seems to get worse the more cores you throw at it. I went ahead and checked out e378601 and it only takes 57 seconds on my 16 core machine, while master takes 240s.

It seems this is a known issue with pytest-xdist:
https://github.com/pytest-dev/pytest-xdist/issues/40
https://github.com/pytest-dev/pytest-xdist/issues/18

I see this all the time for last few months (and mentioned it internally a while ago). I tried to play one day with different scheduling modes (`xdist` has some), but I was never able to get efficient load scheduling.
I'm not sure if the reported pytest-xdist issues are relevant. It looks like too many slow tests are scheduled for certain processes, and other processes are starved.

I have a workaround that seems to improve the runtime to about 81s on master (on the Linux system I mentioned above). It splits the test run into two pytest invocations. I don't love the approach but it's probably worth the performance increase. I'll prepare a PR once I've verified that it works generally.

Resolving patch:
diff --git a/runtests.py b/runtests.py
--- a/runtests.py
+++ b/runtests.py
@@ -14,12 +14,36 @@
     else:
         python_name = 'python3'
 
+# Slow test suites
+CMDLINE = 'PythonCmdline'
+SAMPLES = 'SamplesSuite'
+TYPESHED = 'TypeshedSuite'
+PEP561 = 'TestPEP561'
+EVALUATION = 'PythonEvaluation'
+
+ALL_NON_FAST = [CMDLINE, SAMPLES, TYPESHED, PEP561, EVALUATION]
+
+# We split the pytest run into three parts to improve test
+# parallelization. Each run should have tests that each take a roughly similiar
+# time to run.
 cmds = {
+    # Self type check
     'self': python_name + ' -m mypy --config-file mypy_self_check.ini -p mypy',
+    # Lint
     'lint': 'flake8 -j0',
-    'pytest': 'pytest'
+    # Fast test cases only (this is the bulk of the test suite)
+    'pytest-fast': 'pytest -k "not (%s)"' % ' or '.join(ALL_NON_FAST),
+    # Test cases that invoke mypy (with small inputs)
+    'pytest-cmdline': 'pytest -k "%s"' % ' or '.join([CMDLINE, EVALUATION]),
+    # Test cases that may take seconds to run each
+    'pytest-slow': 'pytest -k "%s"' % ' or '.join([SAMPLES, TYPESHED, PEP561]),
 }
 
+# Stop run immediately if these commands fail
+FAST_FAIL = ['self', 'lint']
+
+assert all(cmd in cmds for cmd in FAST_FAIL)
+
 if not set(args).issubset(cmds):
     print("usage:", prog, " ".join('[%s]' % k for k in cmds))
     exit(1)
@@ -27,9 +51,16 @@
 if not args:
     args = list(cmds)
 
+status = 0
+
 for arg in args:
     cmd = cmds[arg]
-    print('$', cmd)
+    print('run %s: %s' % (arg, cmd))
     res = (system(cmd) & 0x7F00) >> 8
     if res:
-        exit(res)
+        print('\nFAILED: %s' % arg)
+        status = res
+        if arg in FAST_FAIL:
+            exit(status)
+
+exit(status)


Test patch: