Repository: python/mypy

Issue:
mypy 0.982 run 15 minutes (previous version needs only one minute)
**Bug Report**

_My project is not open-source but I can give you (mypy maintainers) access to repository on gitlab._

After updating mypy to newest version it takes 15 minutes to complete on CI and 8 minutes on my PC. When I rollback to previous release it will works fine (about 15 seconds on local machine and about 45 on CI).

I have base class for about 100 subclasses and mypy hangs always in same place.

```
LOG:  Deleting dostanesie.core.formulas.university.ur.base dostanesie/core/formulas/university/ur/base.py dostanesie/core/formulas/university/ur/base.meta.json dostanesie/core/formulas/university/ur/base.data.json
LOG:  Processing SCC singleton (dostanesie.core.formulas.university.uph.base) as inherently stale with stale deps (dostanesie.core.formulas.code)
LOG:  Deleting dostanesie.core.formulas.university.uph.base dostanesie/core/formulas/university/uph/base.py dostanesie/core/formulas/university/uph/base.meta.json dostanesie/core/formulas/university/uph/base.data.json
LOG:  Processing SCC singleton (dostanesie.core.formulas.university.up_wroclaw.base) as inherently stale with stale deps (dostanesie.core.formulas.code)
LOG:  Deleting dostanesie.core.formulas.university.up_wroclaw.base dostanesie/core/formulas/university/up_wroclaw/base.py dostanesie/core/formulas/university/up_wroclaw/base.meta.json dostanesie/core/formulas/university/up_wroclaw/base.data.json
LOG:  Processing SCC singleton (dostanesie.core.formulas.university.up_poznan.base) as inherently stale with stale deps (dostanesie.core.formulas.code)
LOG:  Deleting dostanesie.core.formulas.university.up_poznan.base dostanesie/core/formulas/university/up_poznan/base.py dostanesie/core/formulas/university/up_poznan/base.meta.json dostanesie/core/formulas/university/up_poznan/base.data.json
LOG:  Processing SCC singleton (dostanesie.core.formulas.university.up_lublin.base) as inherently stale with stale deps (dostanesie.core.formulas.code)

```

I tried deleting mypy cache dir locally, on CI we build new docker image each run so there is no mypy cache. 

**To Reproduce**

I really don't know :(

**Expected Behavior**

Mypy should run about one minute for version `0.971`.

```
LOG:  Build finished in 14.859 seconds with 276 modules, and 0 errors
```

**Actual Behavior**

On CI it takes more than 15 minutes, on local machine 8 minutes.


**Your Environment**


- Mypy version used: 0.982
- Mypy command-line flags: none
- Mypy configuration options from `mypy.ini` (and other config files):
```ini
[mypy]
ignore_missing_imports = true
```
- Python version used: Python 3.10.7+



[Full mypy log](https://pastebin.com/EAUt6Bca)



Discussion:
Such things are hard to diagnose. You can try bisecting it to a specific mypy commit, then it will be much easier.
I definitely will try today. Also I will try to extract problematic part of our code and share it on GH
https://github.com/python/mypy/commit/3ae19a25f0a39358ede1383e93d44ef9abf165e0 this takes more than 25 minutes 

I run this in the background and killed after 25 minutes, it takes longer than previous because I use my PC.
I looked at the commit, and a few things seemed potentially suspicious: changes in `sum()`, `subprocess` and `unittest`. If the file where mypy is very slow uses some of these, could you try commenting out the relevant lines to see if it speeds things up?
replaced almost all `sum` with `any` and it takes 30 seconds for commit above (not compiled) and 14 seconds for newest version (compiled). We have something between 100 and 200 `sum` in this package.
That would indicate that the `sum` change is likely the root cause for the regression, since when using `sum` the run was was over 25 minutes. Does this sound right?

In my case: True

we can stick to 0.971, newest version of mypy is not crucial. However, it can affect many other projects CI/CD, especially calculation-dependent (our backend is one big calculator with hundreds calculator classes and some REST API)
Yeah, this seems pretty bad and we should try to fix the regression. 

It would be very helpful if you could you try to narrow this down a bit -- it's possible that there is some specific call or calls to `sum` that are very slow. For example, replace the first 50% of the `sum` calls and then the remaining 50%, in order to narrow things down/bisect (and measure afterwards), and continue until you have a small set of calls that trigger a significant performance regression. This way you may be able to produce a small code example that triggers slow type checking.
@Behoston you could try adding `--timing-stats <path>` to your mypy invocation. It will output a simple text file with per-module timings (one module per line). That should help narrow down which files are most impacted by the change, and from there you could further refine which statements in the specific files are most impacted.
Hello,

We are experiencing similar issues with Colour: We went from ~30mins to ~1h 20mins on CI when updating our dependencies, one of them was mypy==0.961 --> mypy==0.982.

Cheers,

Thomas

Our project also went from ~5min to ~25min going from 0.971 to 0.981/2.
It would be useful if you could post the details of your projects, so we can reproduce the slow down. As is mentioned above, it's very hard to diagnose these things in the abstract.

If your project is not open source, it would be very helpful if you could make a shareable repro of the bad behaviour. For example, try some of the strategies mentioned above:
- Use `--timing-stats` to narrow down which module is slow
- Potentially look for invocations of `sum` (since that's related to OP's problem, lots of thanks to Behoston for narrowing that down)
We can probably just revert the change to sum, it's a somewhat weird change to have: https://github.com/python/mypy/pull/13961

In all cases, would love to have small repros of the performance regression
https://github.com/dostanesie-pl/mypy-bug @hauntsaninja I published part of our code.
Unfortunately I cannot publish our code.
But looking at `--timing-stats` I noticed that in 0.971 all external libs are displayed with "0", while in 0.982 they are ">0".

Maybe that has nothing to do with the cause for this, some of our modules have increased by a factor of 10, while others stay the same.
We do not use pythons builtin `sum` though.
Works much better on version `0.990`
We're still seeing substantial slowdowns on version `0.990`; is there an easy way to see which lines are slow (have already used `--timing-stats` to narrow down the files)
In my case time is acceptable (at least locally, I need to fix some code so
not tested on Ci yet), so I cannot help with 0.990

pt., 11 lis 2022, 14:02 użytkownik James Marchant ***@***.***>
napisał:

> We're still seeing substantial slowdowns on version 0.990; is there an
> easy way to see which lines are slow (have already used --timing-stats to
> narrow down the files)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/python/mypy/issues/13821#issuecomment-1311672499>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB3WCSOZ3NLFXLQB75EGAODWHY7U3ANCNFSM6AAAAAAQ56WG3I>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

0.990 did not really change anything for us, still 1h20+ check times here.
Have bisected the commits between 0.971 and 0.981 to identify which one causes the slowdown for us. For us it seems to be [d27bff62f71a8b914b1df239467148e81d2e88a2](https://github.com/python/mypy/commit/d27bff62f71a8b914b1df239467148e81d2e88a2):

| Commit                                   | No cache runtime | Cached runtime |
|------------------------------------------|------------------|----------------|
| d2063d260ad3096df78867abf53aa247d67b401e |      4m 55.01s     |      3.846s     |
| d27bff62f71a8b914b1df239467148e81d2e88a2 |         Stopped after 1hr 11m         |       N/A      |
Hm, that PR is _almost_ a pure refactor. Actually I have something to suspect. Potentially you have some literal type with a lot of values or some large enum. There is a function `try_contracting_literals_in_union()` that may be computationally intensive, previously it was (inconsistently) only used in `is_subtype()`, after that PR it is used also by `is_proper_subtype()`. But this is just a wild guess. If you don't want to share your code, you will likely need to use `mypy` with a profiler to compare before and after. Note that CProfile will only work with interpreted `mypy`, for compiled version may need to use https://pypi.org/project/py-spy/
I ran `py-spy` on interpreted mypy 0.990 (`mypy 0.990 (compiled: no)`) as slow down is still present in there; it unfortunately isn't a full run as it got to 90 minutes but it looks to capture where it's spending most of the time during that process. I've attached the profiling with personal details removed but it looks like you're right and that `is_subtype()` and `is_proper_subtype()` are potentially the culprits.

![profile](https://user-images.githubusercontent.com/13064654/201898966-37450c9c-3654-4621-a519-a4a5d7ad362e.svg)

For comparison here's the `mypy 0.971 (compiled: no)` profile (only minor changes to address any issues in 0.990 which were mostly now unused `type: ignore`s)

![profile-0 971](https://user-images.githubusercontent.com/13064654/201903119-74a6eac5-0674-438c-8dfe-55d467504832.svg)

@JMMarchant thanks for the data. Unfortunately, I still can't figure out what is going on. Would it be possible to profile on the last good and first bad commit? Just to exclude other possible changes to the call graph (there were quite a lot in recent couple releases). Also could you please post your mypy config?
> @JMMarchant thanks for the data. Unfortunately, I still can't figure out what is going on. Would it be possible to profile on the last good and first bad commit? Just to exclude other possible changes to the call graph (there were quite a lot in recent couple releases). Also could you please post your mypy config?

| Commit                                   | No cache runtime | Cached runtime |
|------------------------------------------|------------------|----------------|
| d2063d260ad3096df78867abf53aa247d67b401e |      [~5m](https://user-images.githubusercontent.com/13064654/202209713-6a342d0d-9e33-4ea7-8713-61e491d08f28.svg)     |     [ ~4s](https://user-images.githubusercontent.com/13064654/202209707-c26292f7-28b4-43ac-b69e-85e915d88e45.svg)     |
| d27bff62f71a8b914b1df239467148e81d2e88a2 |         [~2hrs](https://user-images.githubusercontent.com/13064654/202209678-95acc7d3-33e1-4279-99fe-6934de90ba77.svg)         |       [~4s](https://user-images.githubusercontent.com/13064654/202209669-096b5b0f-10d6-4aff-a273-df8abd6eb12e.svg)      |

`mypy.ini`:
```ini
[mypy]
mypy_path = $MYPY_CONFIG_FILE_DIR/stubs
files =
    src/,
    dev-scripts/,
    docs/,
    scripts/,
    setup.py,
    tests/

; Error message config
show_error_context = True
show_column_numbers = True
show_error_codes = True
pretty = True

; Untyped definitions and calls
disallow_untyped_defs = True
disallow_incomplete_defs = True
check_untyped_defs = True
disallow_untyped_decorators = True
; None and optional
no_implicit_optional = True
; Warnings
warn_redundant_casts = True
warn_unused_ignores = True
warn_no_return = True
warn_return_any = True
warn_unreachable = True

; These are all the third-party libraries we have generated partial or full stubs
; for. We are a bit more permissive for what is allowed in these stubs/how detailed
; they need to be. They can all be found in the stubs/ folder.
;    - albumentations
;    - envyaml
;    - fire
;    - GPUtil
;    - grpc
;    - intermine
;    - lightgbm
;    - marshmallow_enum
;    - marshmallow_union
;    - matplotlib
;    - methodtools
;    - opacus
;    - pandasql
;    - pdoc
;    - pyarrow
;    - pydicom
;    - pytorch_tabnet
;    - scipy
;    - skimage
;    - sklearn
;    - snsql
;    - sphobjinv
;    - sqlparse
;    - sqlvalidator
;    - statsmodels
;    - torchvision
;    - wirerope
[mypy-albumentations.*,envyaml.*,fire.*,GPUtil.*,grpc.*,intermine.*,lightgbm.*,marshmallow_enum.*,marshmallow_union.*,matplotlib.*,methodtools.*,opacus.*,pandasql.*,pdoc.*,pyarrow.*,pydicom.*,pytorch_tabnet.*,scipy.*,skimage.*,sklearn.*,snsql.*,sphobjinv.*,sqlparse.*,sqlvalidator.*,statsmodels.*,torchvision.*,wirerope.*]
allow_incomplete_defs = True
allow_untyped_defs = True
disallow_any_unimported = True


[mypy-docs.*]
disallow_any_expr = True


; Suppress less useful errors for tests
[mypy-tests.*]
; Tests with multiple asserts may cause code to be marked as unreachable even
; though it's fine. See https://github.com/python/mypy/issues/9457
warn_unreachable = False

; Suppress errors for untyped third party test libraries:
; - chromedriver_autoinstaller
; - jupytext
; - nbclient
; - nbformat
; - pytest_flask
; - requests_toolbelt
; - testbook
[mypy-chromedriver_autoinstaller.*,jupytext.*,nbclient.*,nbformat.*,pytest_flask.*,requests_toolbelt.*,testbook.*]
ignore_missing_imports = True
```

@JMMarchant After looking a bit more at these, it seems like something may be slow with union handling and/or overloads. I just merged a PR that adds an optimization to overloads, could you please:
* Check mypy perf on current master
* Try the new internal flag `--line-checking-stats` (also merged today) to see where in your code mypy is stuck
I am also seeing significant performance regression between 0.971 and master: 5min -> 20min for a mypyc-compiled run on my codebase.

The per-file time stats point to a file that has a large function with many if/else branches depending on values of a very large Enum (~13min for a single file).

However the per-line time stats seem to be inconsistent with the per-file stats, and while the same file does show up with some slow lines, the numbers don't add up to the same amount as the per-file time stats. The largest lines on the per-line stats point to overloaded methods taking the enum value as a parameter (specifically `dict.get`), which at that point I expect to be a gigantic `Union` of `Literal` values due to type narrowing in the if branches.

I will try to reduce this file into a more practical reproducer and profile it in the next few days.
Times are expected to not add up exactly because line stats only count time spent in expressions (and only for type checking). There may be some additional time spent analyzing enclosing statement, but it is hard to attribute a line number to this and such time is usually small.
> Times are expected to not add up exactly because line stats only count time spent in expressions (and only for type checking). There may be some additional time spent analyzing enclosing statement, but it is hard to attribute a line number to this and such time is usually small.

I did expect a small difference, but this is not even close:

per-file timing: 790953050us
sum of per-line timings: 1287098.9us

There's a >600x mismatch!
Profiling results are interesting! There are ~20 if statement branches to visit, and each branch spends ~2min (pure-python run) in the `update_from_options` method of `binder.frame_context` after processing the meat of the if statement. Pretty much the entire time is spent in subtype checks.

This also explains the mismatch in timings, as the per-line timings are only collected in `ExpressionChecker`, and not in `TypeChecker`, which is were all of this work is happening.

Relevant speedscope output for a single if statement branch:
<img width="1680" alt="Screen Shot 2022-12-10 at 11 35 36 AM" src="https://user-images.githubusercontent.com/7319881/206872749-304c025d-c8cc-45e7-a4fb-d19705f95af8.png">

@ilevkivskyi I think the merging of subtype visitors is likely to blame for that. I have a hunch that some of the optimizations in the proper subtype check got lost along the way. For instance `_is_proper_subtype` used to have the following fast path, which seems relevant for the kind of large `Union` of `Literal` produced by enum narrowing:

```
    if isinstance(right, UnionType) and not isinstance(left, UnionType):
        return any(
            is_proper_subtype(
                orig_left,
                item,
                ignore_promotions=ignore_promotions,
                erase_instances=erase_instances,
                keep_erased_types=keep_erased_types,
            )
            for item in right.items
        )
```

This sort of issue/regression has come up a number of times. It seems to me it might be worth revisiting the way Enum narrowing is done. Maybe instead of always exploding them into `Union` types and then having to special case those unions in a number of places, only to try to reduce them back to the original Enum later on, we should always keep the enum type itself, and attach extra info to it to indicate relevant narrowing results? Similar to the way `Instance` maintains `last_known_value`
Yep, when you mentioned the mismatch, binder was my first guess (and it indeed uses proper subtypes). I am not sure what is the best way to go here. One option is to restore the previous logic for proper subtypes for unions. Although it would be inconsistent (the whole point of the merge was that proper vs non-proper subtype checks should go through the same code paths except when an `Any` type is involved), this should hopefully fix the perf issue. Having a "proper" fix for enums may take time (and may be also risky).

At least I can try to find a minimal PR that would fix the regression. I will post here when I will have something.
@huguesb @JMMarchant (and others). Could you please try https://github.com/python/mypy/pull/14276 to see if it fixes performance for you?

Resolving patch:
diff --git a/mypy/subtypes.py b/mypy/subtypes.py
--- a/mypy/subtypes.py
+++ b/mypy/subtypes.py
@@ -57,6 +57,7 @@
     UninhabitedType,
     UnionType,
     UnpackType,
+    _flattened,
     get_proper_type,
     is_named_instance,
 )
@@ -891,6 +892,35 @@ def visit_union_type(self, left: UnionType) -> bool:
                 if not self._is_subtype(item, self.orig_right):
                     return False
             return True
+
+        elif isinstance(self.right, UnionType):
+            # prune literals early to avoid nasty quadratic behavior which would otherwise arise when checking
+            # subtype relationships between slightly different narrowings of an Enum
+            # we achieve O(N+M) instead of O(N*M)
+
+            fast_check: set[ProperType] = set()
+
+            for item in _flattened(self.right.relevant_items()):
+                p_item = get_proper_type(item)
+                if isinstance(p_item, LiteralType):
+                    fast_check.add(p_item)
+                elif isinstance(p_item, Instance):
+                    if p_item.last_known_value is None:
+                        fast_check.add(p_item)
+                    else:
+                        fast_check.add(p_item.last_known_value)
+
+            for item in left.relevant_items():
+                p_item = get_proper_type(item)
+                if p_item in fast_check:
+                    continue
+                lit_type = mypy.typeops.simple_literal_type(p_item)
+                if lit_type in fast_check:
+                    continue
+                if not self._is_subtype(item, self.orig_right):
+                    return False
+            return True
+
         return all(self._is_subtype(item, self.orig_right) for item in left.items)
 
     def visit_partial_type(self, left: PartialType) -> bool:
diff --git a/mypy/types.py b/mypy/types.py
--- a/mypy/types.py
+++ b/mypy/types.py
@@ -3343,6 +3343,15 @@ def has_recursive_types(typ: Type) -> bool:
     return typ.accept(_has_recursive_type)
 
 
+def _flattened(types: Iterable[Type]) -> Iterable[Type]:
+    for t in types:
+        tp = get_proper_type(t)
+        if isinstance(tp, UnionType):
+            yield from _flattened(tp.items)
+        else:
+            yield t
+
+
 def flatten_nested_unions(
     types: Iterable[Type], handle_type_alias_type: bool = True
 ) -> list[Type]:


Test patch: