Repository: psf/black

Issue:
Black slow startup to format a single file
I'm a bit annoyed that it takes too much time to format a single file when using black to auto-save contents in my IDE (in this case it's Eclipse/PyDev, but I guess it should be the same for any place using the command line to format a single file).

-- In Eclipse/PyDev, the integration (I've done) is done so that for each new file to be formatted, black is executed and contents are piped in/out.

For doing this execution (in my machine), black takes `0.3s`, but out of that time, `0.255s` (`83%` of the time) is taken on doing imports, `0.025s` (`9%`) getting git info and only `0.020s` (`8%`) on actually doing the formatting...

In use case where a single file is passed is that it'll call `black.files.get_gitignore` even when a single `.py` is passed, which is pretty slow and it shouldn't be needed in this case (without it, the formatting time of `0.045s` goes to `0.02s`... this can be seen passing some dummy `--exclude` which has the side-effect of not loading the git ignore contents).

Some profiling numbers on doing a single run for a file in `black` from the command line:

![image](https://user-images.githubusercontent.com/117621/138691838-64c56d5f-320f-42da-ae2b-c5dc66dbb8d5.png)

So, some things I considered to improve that:

1. Move some of the bigger imports from top-level to inside methods so that it happens lazily (that way things unneeded such as the `asyncio` wouldn't be executed at all). Note: this would need to move some top-level imports to a `typing.TYPE_CHECKING` check too.
2. Make the `get_gitignore` called only if needed.

I guess another option could be using the black server in the integration, but given that it's not installed by default in user environments this is a bit of a hassle for end users (since it's not really polite to automatically mess user environments installing `black[d]` one needs to ask them to do so -- and it doesn't fix the issue where it's used in the command line).

So, I was considering providing a pull request for that -- to make some imports local / don't call `get_gitignore` unless needed -- is this something that seems ok? (for me it's a no-brainer, but I've seen projects adamant on using top-level imports even if it works to their detriment, so, I'd rather ask first than do the work and get rejected later).

Improve black startup to format a single file. Fixes #2564
<!-- Hello! Thanks for submitting a PR. To help make things go a bit more
     smoothly we would appreciate that you go through this template. -->

### Make the startup of black faster when formatting a single file by importing less modules and skipping loading of `.gitignore`.

<!-- Good things to put here include: reasoning for the change (please link
     any relevant issues!), any noteworthy (or hacky) choices to be aware of,
     or what the problem resolved here looked like ... we won't mind a ranty
     story :) -->

With this patch the time to format one file went from a baseline of:
(min /max time from doing many runs on my machine)
`0.29s -> 0.47s` to `0.17s -> 0.32s`

-- Note that this includes the time to format too, not just the startup time. Also, this depends a lot on the target machine (slower machines will probably have more to gain)... in my machine I'd say there's around a 50% increase in the startup speed. There's still room for improvement, but I'd say that the low-hanging fruits have been taken care of.

The number of imported modules for doing a run in this case went from `268` to `202`.

It went something as:

Modules imported (baseline):
268

Without asyncio:
235

Without concurrent.futures
222

Without pathspec:
214

Without multiprocessing:
202



<!-- If any of the following items aren't relevant for your contribution
     please still tick them so we know you've gone through the checklist.

    All user-facing changes should get an entry. Otherwise, signal to us
    this should get the magical label to silence the CHANGELOG entry check.
    Tests are required for bugfixes and new features. Documentation changes
    are necessary for formatting and most enhancement changes. -->

- [x] Add a CHANGELOG entry if necessary?
- [x] Add / update tests if necessary?
- [x] Add new / update outdated documentation?

<!-- Just as a reminder, everyone in all psf/black spaces including PRs
     must follow the PSF Code of Conduct (link below).

     Finally, once again thanks for your time and effort. If you have any
     feedback in regards to your experience contributing here, please
     let us know!

     Helpful links:

      PSF COC: https://www.python.org/psf/conduct/
      Contributing docs: https://black.readthedocs.io/en/latest/contributing/index.html
      Chat on Python Discord: https://discord.gg/RtVdv86PrH -->



Discussion:
Hi! `blackd` is indeed the way to go for repeated small formatting. That being said, I very much appreciate the profiling you've done! Your presented solutions could be worth implementing 👍
Assuming this doesn't require too much refactoring I'm down, asyncio seems like an easy win for one. We already have a work-in-progress solution to tackle the slow formatting bit but targeting the start up time is also great.
Great, I'll take a look into providing a pull request then.
Thanks. I like the idea. I don't know if we (maintainers collective) want lazy imports all over the code base tho. I think it's worth it potentially, but for cases where latency really matters we recommended blackd.

Some of your import moving has broke tests ... I did expect more work to be needed in some of those places. This is the fun and games with moving imports for speed introduces.

Bringing in @ichard26 to ensure this does not break mypyc - I think it's time we add an action to build with mypyc @ichard26 - I'm happy to help here to avoid regressions there.

I am interested to hear others thoughts.
Ok, I've just fixed the tests and added back methods which had been removed from the public API.

Also, lazy imports are added very judiciously in the PR, so, I hope this would be OK (I personally nowadays prefer lazy imports project-wide, but this really needs to be a conscious decision from the project and I understand this isn't the case for black, so, I tried to leave changes to a minimum to make it work yet still bring in substantial benefits for the use case where those modules aren't used at all).

As for the tests which were failing due to the imports, I don't think that something as `black.ProcessPoolExecutor` would be expected as a part of its API, so, fixing the test seems fair... the issue here is more on using monkey-patching for that (it's easy to do, but it really does makes the tests a bit more brittle).

Resolving patch:
diff --git a/src/black/__init__.py b/src/black/__init__.py
--- a/src/black/__init__.py
+++ b/src/black/__init__.py
@@ -1,10 +1,7 @@
-import asyncio
 import io
 import json
-import os
 import platform
 import re
-import signal
 import sys
 import tokenize
 import traceback
@@ -13,10 +10,8 @@
 from datetime import datetime
 from enum import Enum
 from json.decoder import JSONDecodeError
-from multiprocessing import Manager, freeze_support
 from pathlib import Path
 from typing import (
-    TYPE_CHECKING,
     Any,
     Dict,
     Generator,
@@ -38,9 +33,8 @@
 from pathspec.patterns.gitwildmatch import GitWildMatchPatternError
 
 from _black_version import version as __version__
-from black.cache import Cache, filter_cached, get_cache_info, read_cache, write_cache
+from black.cache import Cache, get_cache_info, read_cache, write_cache
 from black.comments import normalize_fmt_off
-from black.concurrency import cancel, maybe_install_uvloop, shutdown
 from black.const import (
     DEFAULT_EXCLUDES,
     DEFAULT_INCLUDES,
@@ -91,9 +85,6 @@
 from blib2to3.pgen2 import token
 from blib2to3.pytree import Leaf, Node
 
-if TYPE_CHECKING:
-    from concurrent.futures import Executor
-
 COMPILED = Path(__file__).suffix in (".pyd", ".so")
 
 # types
@@ -125,8 +116,6 @@ def from_configuration(
 # Legacy name, left for integrations.
 FileMode = Mode
 
-DEFAULT_WORKERS = os.cpu_count()
-
 
 def read_pyproject_toml(
     ctx: click.Context, param: click.Parameter, value: Optional[str]
@@ -375,9 +364,8 @@ def validate_regex(
     "-W",
     "--workers",
     type=click.IntRange(min=1),
-    default=DEFAULT_WORKERS,
-    show_default=True,
-    help="Number of parallel workers",
+    default=None,
+    help="Number of parallel workers [default: number of CPUs in the system]",
 )
 @click.option(
     "-q",
@@ -452,7 +440,7 @@ def main(  # noqa: C901
     extend_exclude: Optional[Pattern[str]],
     force_exclude: Optional[Pattern[str]],
     stdin_filename: Optional[str],
-    workers: int,
+    workers: Optional[int],
     src: Tuple[str, ...],
     config: Optional[str],
 ) -> None:
@@ -588,6 +576,8 @@ def main(  # noqa: C901
                 report=report,
             )
         else:
+            from black.concurrency import reformat_many
+
             reformat_many(
                 sources=sources,
                 fast=fast,
@@ -621,12 +611,7 @@ def get_sources(
 ) -> Set[Path]:
     """Compute the set of files to be formatted."""
     sources: Set[Path] = set()
-
-    if exclude is None:
-        exclude = re_compile_maybe_verbose(DEFAULT_EXCLUDES)
-        gitignore = get_gitignore(ctx.obj["root"])
-    else:
-        gitignore = None
+    root = ctx.obj["root"]
 
     for s in src:
         if s == "-" and stdin_filename:
@@ -661,6 +646,11 @@ def get_sources(
 
             sources.add(p)
         elif p.is_dir():
+            if exclude is None:
+                exclude = re_compile_maybe_verbose(DEFAULT_EXCLUDES)
+                gitignore = get_gitignore(root)
+            else:
+                gitignore = None
             sources.update(
                 gen_python_files(
                     p.iterdir(),
@@ -772,132 +762,6 @@ def reformat_one(
         report.failed(src, str(exc))
 
 
-# diff-shades depends on being to monkeypatch this function to operate. I know it's
-# not ideal, but this shouldn't cause any issues ... hopefully. ~ichard26
-@mypyc_attr(patchable=True)
-def reformat_many(
-    sources: Set[Path],
-    fast: bool,
-    write_back: WriteBack,
-    mode: Mode,
-    report: "Report",
-    workers: Optional[int],
-) -> None:
-    """Reformat multiple files using a ProcessPoolExecutor."""
-    from concurrent.futures import Executor, ProcessPoolExecutor, ThreadPoolExecutor
-
-    executor: Executor
-    worker_count = workers if workers is not None else DEFAULT_WORKERS
-    if sys.platform == "win32":
-        # Work around https://bugs.python.org/issue26903
-        assert worker_count is not None
-        worker_count = min(worker_count, 60)
-    try:
-        executor = ProcessPoolExecutor(max_workers=worker_count)
-    except (ImportError, NotImplementedError, OSError):
-        # we arrive here if the underlying system does not support multi-processing
-        # like in AWS Lambda or Termux, in which case we gracefully fallback to
-        # a ThreadPoolExecutor with just a single worker (more workers would not do us
-        # any good due to the Global Interpreter Lock)
-        executor = ThreadPoolExecutor(max_workers=1)
-
-    loop = asyncio.new_event_loop()
-    asyncio.set_event_loop(loop)
-    try:
-        loop.run_until_complete(
-            schedule_formatting(
-                sources=sources,
-                fast=fast,
-                write_back=write_back,
-                mode=mode,
-                report=report,
-                loop=loop,
-                executor=executor,
-            )
-        )
-    finally:
-        try:
-            shutdown(loop)
-        finally:
-            asyncio.set_event_loop(None)
-        if executor is not None:
-            executor.shutdown()
-
-
-async def schedule_formatting(
-    sources: Set[Path],
-    fast: bool,
-    write_back: WriteBack,
-    mode: Mode,
-    report: "Report",
-    loop: asyncio.AbstractEventLoop,
-    executor: "Executor",
-) -> None:
-    """Run formatting of `sources` in parallel using the provided `executor`.
-
-    (Use ProcessPoolExecutors for actual parallelism.)
-
-    `write_back`, `fast`, and `mode` options are passed to
-    :func:`format_file_in_place`.
-    """
-    cache: Cache = {}
-    if write_back not in (WriteBack.DIFF, WriteBack.COLOR_DIFF):
-        cache = read_cache(mode)
-        sources, cached = filter_cached(cache, sources)
-        for src in sorted(cached):
-            report.done(src, Changed.CACHED)
-    if not sources:
-        return
-
-    cancelled = []
-    sources_to_cache = []
-    lock = None
-    if write_back in (WriteBack.DIFF, WriteBack.COLOR_DIFF):
-        # For diff output, we need locks to ensure we don't interleave output
-        # from different processes.
-        manager = Manager()
-        lock = manager.Lock()
-    tasks = {
-        asyncio.ensure_future(
-            loop.run_in_executor(
-                executor, format_file_in_place, src, fast, mode, write_back, lock
-            )
-        ): src
-        for src in sorted(sources)
-    }
-    pending = tasks.keys()
-    try:
-        loop.add_signal_handler(signal.SIGINT, cancel, pending)
-        loop.add_signal_handler(signal.SIGTERM, cancel, pending)
-    except NotImplementedError:
-        # There are no good alternatives for these on Windows.
-        pass
-    while pending:
-        done, _ = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED)
-        for task in done:
-            src = tasks.pop(task)
-            if task.cancelled():
-                cancelled.append(task)
-            elif task.exception():
-                report.failed(src, str(task.exception()))
-            else:
-                changed = Changed.YES if task.result() else Changed.NO
-                # If the file was written back or was successfully checked as
-                # well-formatted, store this information in the cache.
-                if write_back is WriteBack.YES or (
-                    write_back is WriteBack.CHECK and changed is Changed.NO
-                ):
-                    sources_to_cache.append(src)
-                report.done(src, changed)
-    if cancelled:
-        if sys.version_info >= (3, 7):
-            await asyncio.gather(*cancelled, return_exceptions=True)
-        else:
-            await asyncio.gather(*cancelled, loop=loop, return_exceptions=True)
-    if sources_to_cache:
-        write_cache(cache, sources_to_cache, mode)
-
-
 def format_file_in_place(
     src: Path,
     fast: bool,
@@ -1502,8 +1366,11 @@ def patch_click() -> None:
 
 
 def patched_main() -> None:
-    maybe_install_uvloop()
-    freeze_support()
+    if sys.platform == "win32" and getattr(sys, "frozen", False):
+        from multiprocessing import freeze_support
+
+        freeze_support()
+
     patch_click()
     main()
 
diff --git a/src/black/concurrency.py b/src/black/concurrency.py
--- a/src/black/concurrency.py
+++ b/src/black/concurrency.py
@@ -1,9 +1,26 @@
+"""
+Formatting many files at once via multiprocessing. Contains entrypoint and utilities.
+
+NOTE: this module is only imported if we need to format several files at once.
+"""
+
 import asyncio
 import logging
+import os
+import signal
 import sys
-from typing import Any, Iterable
+from concurrent.futures import Executor, ProcessPoolExecutor, ThreadPoolExecutor
+from multiprocessing import Manager
+from pathlib import Path
+from typing import Any, Iterable, Optional, Set
+
+from mypy_extensions import mypyc_attr
 
+from black import WriteBack, format_file_in_place
+from black.cache import Cache, filter_cached, read_cache, write_cache
+from black.mode import Mode
 from black.output import err
+from black.report import Changed, Report
 
 
 def maybe_install_uvloop() -> None:
@@ -11,7 +28,6 @@ def maybe_install_uvloop() -> None:
 
     This is called only from command-line entry points to avoid
     interfering with the parent process if Black is used as a library.
-
     """
     try:
         import uvloop
@@ -55,3 +71,129 @@ def shutdown(loop: asyncio.AbstractEventLoop) -> None:
         cf_logger = logging.getLogger("concurrent.futures")
         cf_logger.setLevel(logging.CRITICAL)
         loop.close()
+
+
+# diff-shades depends on being to monkeypatch this function to operate. I know it's
+# not ideal, but this shouldn't cause any issues ... hopefully. ~ichard26
+@mypyc_attr(patchable=True)
+def reformat_many(
+    sources: Set[Path],
+    fast: bool,
+    write_back: WriteBack,
+    mode: Mode,
+    report: Report,
+    workers: Optional[int],
+) -> None:
+    """Reformat multiple files using a ProcessPoolExecutor."""
+    maybe_install_uvloop()
+
+    executor: Executor
+    if workers is None:
+        workers = os.cpu_count() or 1
+    if sys.platform == "win32":
+        # Work around https://bugs.python.org/issue26903
+        workers = min(workers, 60)
+    try:
+        executor = ProcessPoolExecutor(max_workers=workers)
+    except (ImportError, NotImplementedError, OSError):
+        # we arrive here if the underlying system does not support multi-processing
+        # like in AWS Lambda or Termux, in which case we gracefully fallback to
+        # a ThreadPoolExecutor with just a single worker (more workers would not do us
+        # any good due to the Global Interpreter Lock)
+        executor = ThreadPoolExecutor(max_workers=1)
+
+    loop = asyncio.new_event_loop()
+    asyncio.set_event_loop(loop)
+    try:
+        loop.run_until_complete(
+            schedule_formatting(
+                sources=sources,
+                fast=fast,
+                write_back=write_back,
+                mode=mode,
+                report=report,
+                loop=loop,
+                executor=executor,
+            )
+        )
+    finally:
+        try:
+            shutdown(loop)
+        finally:
+            asyncio.set_event_loop(None)
+        if executor is not None:
+            executor.shutdown()
+
+
+async def schedule_formatting(
+    sources: Set[Path],
+    fast: bool,
+    write_back: WriteBack,
+    mode: Mode,
+    report: "Report",
+    loop: asyncio.AbstractEventLoop,
+    executor: "Executor",
+) -> None:
+    """Run formatting of `sources` in parallel using the provided `executor`.
+
+    (Use ProcessPoolExecutors for actual parallelism.)
+
+    `write_back`, `fast`, and `mode` options are passed to
+    :func:`format_file_in_place`.
+    """
+    cache: Cache = {}
+    if write_back not in (WriteBack.DIFF, WriteBack.COLOR_DIFF):
+        cache = read_cache(mode)
+        sources, cached = filter_cached(cache, sources)
+        for src in sorted(cached):
+            report.done(src, Changed.CACHED)
+    if not sources:
+        return
+
+    cancelled = []
+    sources_to_cache = []
+    lock = None
+    if write_back in (WriteBack.DIFF, WriteBack.COLOR_DIFF):
+        # For diff output, we need locks to ensure we don't interleave output
+        # from different processes.
+        manager = Manager()
+        lock = manager.Lock()
+    tasks = {
+        asyncio.ensure_future(
+            loop.run_in_executor(
+                executor, format_file_in_place, src, fast, mode, write_back, lock
+            )
+        ): src
+        for src in sorted(sources)
+    }
+    pending = tasks.keys()
+    try:
+        loop.add_signal_handler(signal.SIGINT, cancel, pending)
+        loop.add_signal_handler(signal.SIGTERM, cancel, pending)
+    except NotImplementedError:
+        # There are no good alternatives for these on Windows.
+        pass
+    while pending:
+        done, _ = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED)
+        for task in done:
+            src = tasks.pop(task)
+            if task.cancelled():
+                cancelled.append(task)
+            elif task.exception():
+                report.failed(src, str(task.exception()))
+            else:
+                changed = Changed.YES if task.result() else Changed.NO
+                # If the file was written back or was successfully checked as
+                # well-formatted, store this information in the cache.
+                if write_back is WriteBack.YES or (
+                    write_back is WriteBack.CHECK and changed is Changed.NO
+                ):
+                    sources_to_cache.append(src)
+                report.done(src, changed)
+    if cancelled:
+        if sys.version_info >= (3, 7):
+            await asyncio.gather(*cancelled, return_exceptions=True)
+        else:
+            await asyncio.gather(*cancelled, loop=loop, return_exceptions=True)
+    if sources_to_cache:
+        write_cache(cache, sources_to_cache, mode)


Test patch:
diff --git a/tests/test_black.py b/tests/test_black.py
--- a/tests/test_black.py
+++ b/tests/test_black.py
@@ -1757,7 +1757,9 @@ def test_output_locking_when_writeback_diff(self, color: bool) -> None:
                 src = (workspace / f"test{tag}.py").resolve()
                 with src.open("w") as fobj:
                     fobj.write("print('hello')")
-            with patch("black.Manager", wraps=multiprocessing.Manager) as mgr:
+            with patch(
+                "black.concurrency.Manager", wraps=multiprocessing.Manager
+            ) as mgr:
                 cmd = ["--diff", str(workspace)]
                 if color:
                     cmd.append("--color")
@@ -1804,7 +1806,7 @@ def test_filter_cached(self) -> None:
                 str(cached): black.get_cache_info(cached),
                 str(cached_but_changed): (0.0, 0),
             }
-            todo, done = black.filter_cached(
+            todo, done = black.cache.filter_cached(
                 cache, {uncached, cached, cached_but_changed}
             )
             assert todo == {uncached, cached_but_changed}