Repository: python/mypy

Issue:
runtests.py looks like it's hanging at first (though it's not)
Starting our test suite, it looks like this and stays this way for many seconds:
```
$ time ./runtests.py 
PARALLEL 4
SUMMARY  151 tasks selected
passed 0, failed 0, pending 150; running 1                                      
```

When @nurdok saw this on first trying to run tests on his Windows laptop, he figured the tests were hanging -- and when I first looked at it with him, I thought the same thing too.

There are probably two confusing things here:
* The display doesn't change for a long time, with no progress indication.
* The output says "PARALLEL 4", but then there's only 1 task "running", not 4.

Probably the first of these is most important.

The root cause of both issues is that that 1 "task" which is running is actually a `py.test` process, which is running many test cases and doing its own parallelism among those -- but neither the parallelism nor the ongoing progress of finishing those test cases is being reported. Before #3019 we ran the big `py.test` process in parallel with other tasks, some of which finished quickly, which avoided both of these.

The real fix for this will be to complete the migration from our custom `runtests.py` runner to plain `py.test`. In the meantime, we should do something simple to make this not confusing.



Discussion:
A simple fix might be to count the pytest run as 4 tasks (or whatever level of parallelism we are using).
Yeah -- though that wouldn't help with the "nothing changes for a long time" problem, which I think is probably the bigger one.

A simple fix that would also address that part would be to revert the bits of #3019 that added a `sequential` parameter and started using that -- so that we run pytest alongside three other tests. Not optimal for performance, but I'm not sure the difference will be a big one. The way I'd start is to try that, time it before and after, and see if it's OK.

I wonder if the README should focus more on running pytest and less on runtests.py. That's the advice I give beginners. pytest is something they already know, and it's easy to select tests and to run pdb on tests (once you have discovered `pytest -n0`).
We should probably also mention `pytest -n0` in the docs.
Agreed on all counts re the README. I'll open an issue or two for those.

I thought @JukkaL  didn't want to run pytest concurrently with other tests because of the very large peak memory usage. 

Other than that, the performance impact is not necessarily on the total time, but on the responsiveness of the system while the tests are running: with too many CPU-intensive tasks running concurrently, Windows lags on any user operations. I'm guessing Linux handles it better?

Regardless, do we want to pass through pytest progress bar to the console?

For what it's worth, I've also noticed Mac OS becoming unresponsive while I run the mypy test suite.
@pkch Earlier we had multiple pytest runs running in parallel (each with multiple processes) and that caused Travis build failures due to very high memory use. Just running a single parallel pytest run together with other, single-process tasks might work, though I share the concern that running many processes will likely cause system unresponsiveness.

Resolving patch:
diff --git a/mypy/waiter.py b/mypy/waiter.py
--- a/mypy/waiter.py
+++ b/mypy/waiter.py
@@ -25,16 +25,20 @@ class LazySubprocess:
     """Wrapper around a subprocess that runs a test task."""
 
     def __init__(self, name: str, args: List[str], *, cwd: str = None,
-                 env: Dict[str, str] = None) -> None:
+                 env: Dict[str, str] = None, passthrough: bool = False) -> None:
         self.name = name
         self.args = args
         self.cwd = cwd
         self.env = env
         self.start_time = None  # type: float
         self.end_time = None  # type: float
+        self.passthrough = passthrough
 
     def start(self) -> None:
-        self.outfile = tempfile.TemporaryFile()
+        if self.passthrough:
+            self.outfile = None
+        else:
+            self.outfile = tempfile.TemporaryFile()
         self.start_time = time.perf_counter()
         self.process = Popen(self.args, cwd=self.cwd, env=self.env,
                              stdout=self.outfile, stderr=STDOUT)
@@ -47,6 +51,8 @@ def status(self) -> Optional[int]:
         return self.process.returncode
 
     def read_output(self) -> str:
+        if self.passthrough:
+            return ''
         file = self.outfile
         file.seek(0)
         # Assume it's ascii to avoid unicode headaches (and portability issues).
diff --git a/runtests.py b/runtests.py
--- a/runtests.py
+++ b/runtests.py
@@ -111,7 +111,8 @@ def add_pytest(self, name: str, pytest_args: List[str], coverage: bool = False)
         else:
             args = [sys.executable, '-m', 'pytest'] + pytest_args
 
-        self.waiter.add(LazySubprocess(full_name, args, env=self.env), sequential=True)
+        self.waiter.add(LazySubprocess(full_name, args, env=self.env, passthrough=True),
+                        sequential=True)
 
     def add_python(self, name: str, *args: str, cwd: Optional[str] = None) -> None:
         name = 'run %s' % name


Test patch: