Repository: python/mypy
Pull request: Fix daemon crash on malformed NamedTuple
Fixes #14098 

Having invalid statements in a NamedTuple is almost like a syntax error, we can remove them after giving an error (without further analysis). This PR does almost exactly the same as https://github.com/python/mypy/pull/13963 did for TypedDicts.

Problem statement:
Mypy Deamon crash on second run
<!--
  Use this form only if mypy reports an "INTERNAL ERROR" and/or gives a traceback.
  Please include the traceback and all other messages below (use `mypy --show-traceback`).
-->

**Crash Report**
I saw this crash when using Mypy extension for VS Code, but it is reproducible in my code base even from command line by directly invoking `dmypy` as a module. The crash is in two phases:

1. In the first run, the deamon correctly reports errors: the final message is `Found 238 errors in 46 files (checked 1125 source files)`, then the deamon exits (I don't know if exiting in this situation is the expected behaviour).
2. When I try to restart the deamon the following traceback appears.

Each time I run the command, these two phases alternate.

**Traceback**

```
Daemon crashed!
Traceback (most recent call last):
  File "mypy/dmypy_server.py", line 227, in serve
  File "mypy/dmypy_server.py", line 270, in run_command
  File "mypy/dmypy_server.py", line 336, in cmd_run
  File "mypy/dmypy_server.py", line 405, in check
  File "mypy/dmypy_server.py", line 589, in fine_grained_increment_follow_imports
  File "mypy/server/update.py", line 277, in update
  File "mypy/server/update.py", line 862, in propagate_changes_using_dependencies
  File "mypy/server/update.py", line 996, in reprocess_nodes
  File "mypy/server/astmerge.py", line 140, in merge_asts
  File "mypy/server/astmerge.py", line 192, in replace_nodes_in_ast
  File "mypy/nodes.py", line 374, in accept
  File "mypy/server/astmerge.py", line 213, in visit_mypy_file
  File "mypy/traverser.py", line 114, in visit_mypy_file
  File "mypy/nodes.py", line 1127, in accept
  File "mypy/server/astmerge.py", line 238, in visit_class_def
  File "mypy/server/astmerge.py", line 393, in process_synthetic_type_info
  File "mypy/nodes.py", line 208, in accept
RuntimeError: Not implemented
```

**To Reproduce**

Unfortunately I cannot share code for now, I will try to make a reproducible example later. For now I post this error to see if someone can direct me to further steps to better understand the root of the problem or point out trivial mistakes or misconfigurations.

**Your Environment**

<!-- Include as many relevant details about the environment you experienced the bug in -->

- Mypy version used: dmypy version 0.982
- Mypy command-line flags: the following is the command executed by the VS Code extension and then executed by me afterwards:
```shell
$path_to_python -m mypy.dmypy \
    --status-file /home/.../filename.json \
    run \
    --log-file /home/.../filename.log \
    -- $project_directory \
    --show-traceback --show-column-numbers --python-executable $path_to_python
```
- Mypy configuration options from `mypy.ini` (and other config files):
```ini
[mypy]
warn_return_any = True
warn_unused_configs = True
ignore_missing_imports = True
explicit_package_bases = True
namespace_packages = True
```
- Python version used: **Python 3.9.6**
- Operating system and version: **Ubuntu 18.04**

<!--
You can freely edit this text, please remove all the lines
you believe are unnecessary.
-->

Hints from issue discussion:
Thanks for reporting! Hard to figure out without a repro. From the stacktrace I suspect it's something to do with namedtuples, in case that helps narrow things down.

You could try installing mypy master and seeing if it repros. Once you've installed mypy master you could also put in a breakpoint and see exactly what class definition it's struggling with.
Hi! Yes I know, without code this is almost nonsense. Thanks for the suggestion, as soon as I can I will install mypy master and try to debug! I will report back if I find something. 
I have a repro
```
[case testNamedTupleNestedCrash]
import m
[file m.py]
from typing import NamedTuple

class NT(NamedTuple):
    class C: ...
    x: int
    y: int

[file m.py.2]
from typing import NamedTuple

class NT(NamedTuple):
    class C: ...
    x: int
    y: int
# change
[builtins fixtures/tuple.pyi]
```
crashes with the same traceback (in cached mode). Although @ivan94fi please still check your code, maybe you have a different scenario. If it is the same, I think I have a simple fix for this crash, similar to what @Michael0x2a did for TypedDicts.
Hi everyone, I tested mypy master and I can reproduce the crash. However I noticed that the crash only happens when using the dmypy daemon, non when running mypy directly (both for the master and the 0.982 version).

I tried to put a breakpoint in the affected code but I always get a `bdb.BdbQuit` and cannot perform interactive debugging. I suppose because the code is executed by the daemon. I added the `--pdb` flag at the end of the command but this did not change anything.

I tried to put a breakpoint in the following files, in many places, but I always got the above behaviour:
* [dmypy_server.py](https://github.com/python/mypy/blob/7d0d1d9d505869d55ee71e370941a8a47122087e/mypy/dmypy_server.py): here I tried inside  `Server.serve` method, in various lines, both inside the try catches, and outside
* [dmypy/\_\_main\_\_.py](https://github.com/python/mypy/blob/7d0d1d9d505869d55ee71e370941a8a47122087e/mypy/dmypy/__main__.py): here i put a breakpoint before `console_entry`, and then continued, just to be sure to be inside the debugger

Can someone suggest me where to put a breakpoint? Thank you.
You should try putting a breakpoint _before_ this line https://github.com/python/mypy/blob/master/mypy/server/astmerge.py#L393 (also maybe try using `pdb`, I never tried `bdb`). If this still will not work, you can simply add a print statement before that line, but then you will need to pass some file to `--log-file` when starting daemon (it doesn't print to stdout). And check tail of that file when daemon crashes.
Ok I will try as soon as possible, thank you.

By the way, I used `pdb` too, but that exception resides in `bdb`, which apparently is the package containing the basic functions of `pdb`.

Hi I managed to track down the file that was causing the issue and it has the same exact structure of the example reported by @ilevkivskyi 3 comments back, that is, a `NamedTuple` with an inner class:
```python
from typing import Dict, NamedTuple, List

# Class and attributes names are changed, the structure is identical to the one in the original code
class A(NamedTuple):
    a: List[str]
    b: Dict
    c: str
    d: List[str]

    class AInner:
        e = False
```

It is interesting to note that executing mypy (both daemon and cli) on the above code, does not make mypy crash but it returns an error:
`mypy_crash.py:10: error: Invalid statement in NamedTuple definition; expected "field_name: field_type [= default]"`

However removing this code from the original project where it provoked a mypy crash, makes mypy stop crashing.
OK, thanks for confirming! I will try to find time for this in coming days (unless someone else does it first).

Commit message: Fix daemon crash on malformed NamedTuple

Patch to review:
diff --git a/mypy/semanal_namedtuple.py b/mypy/semanal_namedtuple.py
index 04308db99e63..27f664fc0c88 100644
--- a/mypy/semanal_namedtuple.py
+++ b/mypy/semanal_namedtuple.py
@@ -32,6 +32,7 @@
     NameExpr,
     PassStmt,
     RefExpr,
+    Statement,
     StrExpr,
     SymbolTable,
     SymbolTableNode,
@@ -111,7 +112,7 @@ def analyze_namedtuple_classdef(
                     if result is None:
                         # This is a valid named tuple, but some types are incomplete.
                         return True, None
-                    items, types, default_items = result
+                    items, types, default_items, statements = result
                     if is_func_scope and "@" not in defn.name:
                         defn.name += "@" + str(defn.line)
                     existing_info = None
@@ -123,6 +124,7 @@ def analyze_namedtuple_classdef(
                     defn.analyzed = NamedTupleExpr(info, is_typed=True)
                     defn.analyzed.line = defn.line
                     defn.analyzed.column = defn.column
+                    defn.defs.body = statements
                     # All done: this is a valid named tuple with all types known.
                     return True, info
         # This can't be a valid named tuple.
@@ -130,24 +132,27 @@ def analyze_namedtuple_classdef(
 
     def check_namedtuple_classdef(
         self, defn: ClassDef, is_stub_file: bool
-    ) -> tuple[list[str], list[Type], dict[str, Expression]] | None:
+    ) -> tuple[list[str], list[Type], dict[str, Expression], list[Statement]] | None:
         """Parse and validate fields in named tuple class definition.
 
         Return a three tuple:
           * field names
           * field types
           * field default values
+          * valid statements
         or None, if any of the types are not ready.
         """
         if self.options.python_version < (3, 6) and not is_stub_file:
             self.fail("NamedTuple class syntax is only supported in Python 3.6", defn)
-            return [], [], {}
+            return [], [], {}, []
         if len(defn.base_type_exprs) > 1:
             self.fail("NamedTuple should be a single base", defn)
         items: list[str] = []
         types: list[Type] = []
         default_items: dict[str, Expression] = {}
+        statements: list[Statement] = []
         for stmt in defn.defs.body:
+            statements.append(stmt)
             if not isinstance(stmt, AssignmentStmt):
                 # Still allow pass or ... (for empty namedtuples).
                 if isinstance(stmt, PassStmt) or (
@@ -160,9 +165,11 @@ def check_namedtuple_classdef(
                 # And docstrings.
                 if isinstance(stmt, ExpressionStmt) and isinstance(stmt.expr, StrExpr):
                     continue
+                statements.pop()
                 self.fail(NAMEDTUP_CLASS_ERROR, stmt)
             elif len(stmt.lvalues) > 1 or not isinstance(stmt.lvalues[0], NameExpr):
                 # An assignment, but an invalid one.
+                statements.pop()
                 self.fail(NAMEDTUP_CLASS_ERROR, stmt)
             else:
                 # Append name and type in this case...
@@ -199,7 +206,7 @@ def check_namedtuple_classdef(
                         )
                 else:
                     default_items[name] = stmt.rvalue
-        return items, types, default_items
+        return items, types, default_items, statements
 
     def check_namedtuple(
         self, node: Expression, var_name: str | None, is_func_scope: bool
diff --git a/test-data/unit/check-class-namedtuple.test b/test-data/unit/check-class-namedtuple.test
index 8e0545953bd8..8ae7f6555f9d 100644
--- a/test-data/unit/check-class-namedtuple.test
+++ b/test-data/unit/check-class-namedtuple.test
@@ -393,8 +393,6 @@ class X(typing.NamedTuple):
 [out]
 main:6: error: Invalid statement in NamedTuple definition; expected "field_name: field_type [= default]"
 main:7: error: Invalid statement in NamedTuple definition; expected "field_name: field_type [= default]"
-main:7: error: Type cannot be declared in assignment to non-self attribute
-main:7: error: "int" has no attribute "x"
 main:9: error: Non-default NamedTuple fields cannot follow default fields
 
 [builtins fixtures/list.pyi]
diff --git a/test-data/unit/fine-grained.test b/test-data/unit/fine-grained.test
index a6d8f206fbba..201fb1037e30 100644
--- a/test-data/unit/fine-grained.test
+++ b/test-data/unit/fine-grained.test
@@ -10205,3 +10205,27 @@ C
 [builtins fixtures/dict.pyi]
 [out]
 ==
+
+[case testNamedTupleNestedCrash]
+import m
+[file m.py]
+from typing import NamedTuple
+
+class NT(NamedTuple):
+    class C: ...
+    x: int
+    y: int
+
+[file m.py.2]
+from typing import NamedTuple
+
+class NT(NamedTuple):
+    class C: ...
+    x: int
+    y: int
+# change
+[builtins fixtures/tuple.pyi]
+[out]
+m.py:4: error: Invalid statement in NamedTuple definition; expected "field_name: field_type [= default]"
+==
+m.py:4: error: Invalid statement in NamedTuple definition; expected "field_name: field_type [= default]"


Code review comments:
diff_hunk: @@ -123,31 +124,35 @@ def analyze_namedtuple_classdef(
                     defn.analyzed = NamedTupleExpr(info, is_typed=True)
                     defn.analyzed.line = defn.line
                     defn.analyzed.column = defn.column
+                    defn.defs.body = statements
                     # All done: this is a valid named tuple with all types known.
                     return True, info
         # This can't be a valid named tuple.
         return False, None
 
     def check_namedtuple_classdef(
         self, defn: ClassDef, is_stub_file: bool
-    ) -> tuple[list[str], list[Type], dict[str, Expression]] | None:
+    ) -> tuple[list[str], list[Type], dict[str, Expression], list[Statement]] | None:
         """Parse and validate fields in named tuple class definition.
 
         Return a three tuple:
path: mypy/semanal_namedtuple.py
text: @user1:
```suggestion
        Return a four tuple:
```