Repository: python/mypy

Issue:
Internal Error on complex lambda expression
```
~/s/px (python|●2✚1) $ mypy --show-traceback --check-untyped-defs --ignore-missing-imports *.py */*.py
px/px_terminal.py:23: error: Incompatible types in assignment (expression has type "int", variable has type "str")
px/px_terminal.py:24: error: Unsupported operand types for > ("int" and "str")
px/px_terminal.py:29: error: Incompatible types in assignment (expression has type "int", variable has type "str")
px/px_terminal.py:30: error: Unsupported operand types for > ("int" and "str")
px/px_ipc_map.py:45: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.511
Traceback (most recent call last):
  File "/usr/local/bin/mypy", line 6, in <module>
    main(__file__)
  File "/usr/local/lib/python3.6/site-packages/mypy/main.py", line 46, in main
    res = type_check_only(sources, bin_dir, options)
  File "/usr/local/lib/python3.6/site-packages/mypy/main.py", line 93, in type_check_only
    options=options)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 188, in build
    graph = dispatch(sources, manager)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 1595, in dispatch
    process_graph(graph, manager)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 1838, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 1942, in process_stale_scc
    if graph[id].type_check_second_pass():
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 1516, in type_check_second_pass
    return self.type_checker.check_second_pass()
  File "/usr/local/lib/python3.6/site-packages/mypy/checker.py", line 219, in check_second_pass
    self.check_partial(node)
  File "/usr/local/lib/python3.6/site-packages/mypy/checker.py", line 226, in check_partial
    self.expr_checker.accept(node)
  File "/usr/local/lib/python3.6/site-packages/mypy/checkexpr.py", line 2058, in accept
    typ = node.accept(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/nodes.py", line 1548, in accept
    return visitor.visit_lambda_expr(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/checkexpr.py", line 1792, in visit_lambda_expr
    ret_type = self.accept(e.expr(), allow_none_return=True)
  File "/usr/local/lib/python3.6/site-packages/mypy/checkexpr.py", line 2058, in accept
    typ = node.accept(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/nodes.py", line 1451, in accept
    return visitor.visit_op_expr(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/checkexpr.py", line 1137, in visit_op_expr
    return self.check_boolean_op(e, e)
  File "/usr/local/lib/python3.6/site-packages/mypy/checkexpr.py", line 1370, in check_boolean_op
    right_type = self.analyze_cond_branch(right_map, e.right, left_type)
  File "/usr/local/lib/python3.6/site-packages/mypy/checkexpr.py", line 2025, in analyze_cond_branch
    with self.chk.binder.frame_context(can_skip=True, fall_through=0):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 82, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.6/site-packages/mypy/binder.py", line 312, in frame_context
    assert len(self.frames) > 1
AssertionError:
px/px_ipc_map.py:45: note: use --pdb to drop into pdb
~/s/px (python|●2✚1) $
```


Discussion:
Does it still occur with master? Can you show the source code?
Hm, I am unable to reproduce this. I cloned https://github.com/walles/px, and did the following:
With mypy=0.511:
```
$mypy --show-traceback --check-untyped-defs --ignore-missing-imports px
px\px_terminal.py:23: error: Incompatible types in assignment (expression has type "int", variable has type "str")
px\px_terminal.py:24: error: Unsupported operand types for > ("int" and "str")
px\px_terminal.py:29: error: Incompatible types in assignment (expression has type "int", variable has type "str")
px\px_terminal.py:30: error: Unsupported operand types for > ("int" and "str")
$cd px
$mypy --show-traceback --check-untyped-defs --ignore-missing-imports *.py */*.py
mypy: can't read file '*.py': Invalid argument
$mypy --show-traceback --check-untyped-defs --ignore-missing-imports px.py px_commandline.py px_cpuinfo.py px_file.py px_install.py px_ipc_map.py px_load.py px_load_bar.py px_loginhistory.py px_process.py px_processinfo.py px_terminal.py px_top.py __init__.py
Traceback (most recent call last):
  File "C:\Python36\Scripts\mypy", line 6, in <module>
    main(__file__)
  File "C:\Python36\lib\site-packages\mypy\main.py", line 46, in main
    res = type_check_only(sources, bin_dir, options)
  File "C:\Python36\lib\site-packages\mypy\main.py", line 93, in type_check_only
    options=options)
  File "C:\Python36\lib\site-packages\mypy\build.py", line 188, in build
    graph = dispatch(sources, manager)
  File "C:\Python36\lib\site-packages\mypy\build.py", line 1587, in dispatch
    graph = load_graph(sources, manager)
  File "C:\Python36\lib\site-packages\mypy\build.py", line 1696, in load_graph
    ancestor_for=st)
  File "C:\Python36\lib\site-packages\mypy\build.py", line 1166, in __init__
    assert id or path or source is not None, "Neither id, path nor source given"
AssertionError: Neither id, path nor source given
```

I also ran `mypy --show-traceback --check-untyped-defs --ignore-missing-imports px` with master and it was exactly the same as 0.511.

Hm, you shoudn't be running mypy from *inside* a package. Maybe the OP's
exception is because he's passing the files in a different order?

Tar file of what I have right now:
https://drive.google.com/file/d/0Byagq3owDalGVXh1UlBkOHVIcHM/view?usp=sharing

Then `cd` into there and do:
```
mypy --show-traceback --check-untyped-defs --ignore-missing-imports *.py */*.py
```
I'm using the `fish` shell, here's the file order:
```
~/s/px (python|●2✚1) $ echo *.py */*.py
setup.py px/__init__.py px/px.py px/px_commandline.py px/px_cpuinfo.py px/px_file.py px/px_install.py px/px_ipc_map.py px/px_load.py px/px_load_bar.py px/px_loginhistory.py px/px_process.py px/px_processinfo.py px/px_terminal.py px/px_top.py tests/benchmark_ipcmap.py tests/px_commandline_test.py tests/px_cpuinfo_test.py tests/px_file_test.py tests/px_ipc_map_test.py tests/px_load_bar_test.py tests/px_load_test.py tests/px_loginhistory_test.py tests/px_process_test.py tests/px_processinfo_test.py tests/px_terminal_test.py tests/px_top_test.py tests/testutils.py
~/s/px (python|●2✚1) $
```
Hm, I still am unable to repro it. This is the same as the tip of the `python` branch, yes?

```
$python3 -m mypy --show-traceback --check-untyped-defs --ignore-missing-imports setup.py px/__init__.py px/px.py px/px_commandline.py px/px_cpuinfo.py px/px_file.py px/px_install.py px/px_ipc_map.py px/px_load.py px/px_load_bar.py px/px_loginhistory.py px/px_process.py px/px_processinfo.py px/px_terminal.py px/px_top.py tests/benchmark_ipcmap.py tests/px_commandline_test.py tests/px_cpuinfo_test.py tests/px_file_test.py tests/px_ipc_map_test.py tests/px_load_bar_test.py tests/px_load_test.py tests/px_loginhistory_test.py tests/
px_process_test.py tests/px_processinfo_test.py tests/px_terminal_test.py tests/px_top_test.py tests/testutils.py
px\px_terminal.py:23: error: Incompatible types in assignment (expression has type "int", variable has type "str")
px\px_terminal.py:24: error: Unsupported operand types for > ("int" and "str")
px\px_terminal.py:29: error: Incompatible types in assignment (expression has type "int", variable has type "str")
px\px_terminal.py:30: error: Unsupported operand types for > ("int" and "str")
```
EDIT: I also installed and ran from within fish, and _still_ couldn't repro it.
Nope, I have some changes, this is not the same as the tip.

I also have a `.mypy_cache` in the tar file, which could be involved I guess.
Here's the diff between `HEAD` of the `python` branch and what's in my working dir:
```diff
~/s/px (python|●2✚1) $ git diff HEAD|cat
diff --git a/px/px_ipc_map.py b/px/px_ipc_map.py
index 4e73f6c..34b6e77 100644
--- a/px/px_ipc_map.py
+++ b/px/px_ipc_map.py
@@ -40,8 +40,9 @@ class IpcMap(object):
         # process. Putting the files in a set gives us each file only once.
         files = set(files)

-        # Only deal with IPC related files
-        self.files = list(filter(lambda f: f.type in FILE_TYPES, files))
+        # Only deal with IPC related files and files for our own process
+        self.files = list(filter(
+            lambda f: (f.type in FILE_TYPES) or (f.pid == self.process.pid), files))

         self.process = process
         self.processes = processes
@@ -50,6 +51,10 @@ class IpcMap(object):
         self._map = {}  # type: MutableMapping[px_process.PxProcess, Set[px_file.PxFile]]
         self._create_mapping()

+        self.stdin = None  # type: str
+        self.stdout = None  # type: str
+        self.stderr = None  # type: str
+
     def _create_mapping(self):
         # type: () -> None
         self._create_indices()
diff --git a/tests/px_ipc_map_test.py b/tests/px_ipc_map_test.py
index 8a25a83..a8cd873 100644
--- a/tests/px_ipc_map_test.py
+++ b/tests/px_ipc_map_test.py
@@ -169,3 +169,15 @@ def test_get_ipc_map_2():

     peer0 = keys[0]
     assert len(ipc_map[peer0]) == 4
+
+
+def test_stdfd_ipc():
+    files = None
+    my_dir = os.path.dirname(__file__)
+    with open(os.path.join(my_dir, "lsof-test-output-linux-1.txt"), "r") as lsof_output:
+        files = px_file.lsof_to_files(lsof_output.read())
+
+    ipc_map = testutils.create_ipc_map(287, files)
+    assert ipc_map.stdin == 'FIXME: A reference to the other process'
+    assert ipc_map.stdout == '/dev/null'
+    assert ipc_map.stderr == 'FIXME: A reference to the other process'
```

Repro'd. It is (as I suspected) the lambda statement (making _just_ that change triggers the crash). I will look at the cause. Thanks for reporting!
```
from typing import List

FILE_TYPES  # type: List[str]

class PxProcess:
    pid: int
    
class File:
    type: str
    pid: int
    
class IpcMap(object):
    
    def __init__(self, files: List[File], process: PxProcess) -> None:
        self.files = list(filter(
            lambda f: (f.type in FILE_TYPES) or (f.pid == self.process.pid), files))
        self.process = process
```
This seems to be an issue where Mypy fails to check things correctly for `self.process.pid`. @walles you should probably put this statement _after_ assigning `process` to `self.process`, so there is a programmer error here, but also a bug in MyPy of course. 
If you move `self.process = process` before the assignment to `self.files` things work.

This bug _is_ present in master.
(Removing the 'bug' label because the 'crash' label implies it.)
The crash reported by @ethanhs ("AssertionError: Neither id, path nor source given") is a dupe of #2974. But the crash reported by the OP (and later also reproduced by Ethan) is not, so don't close this.

I'm also getting this with what I presume is a complex lambda expression. I haven't been able to make a smaller reproduction, but here's the error from running on my codebase:

```
$ mypy src/ --verbose --show-traceback 

...

Traceback (most recent call last):
  File "/home/daboross/Projects/Python/Screeps/env/bin/mypy", line 11, in <module>
    sys.exit(console_entry())
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/main.py", line 97, in type_check_only
    options=options)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/build.py", line 196, in build
    graph = dispatch(sources, manager)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/build.py", line 1801, in dispatch
    process_graph(graph, manager)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/build.py", line 2044, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/build.py", line 2152, in process_stale_scc
    if graph[id].type_check_second_pass():
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/build.py", line 1722, in type_check_second_pass
    return self.type_checker.check_second_pass()
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checker.py", line 228, in check_second_pass
    self.check_partial(node)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checker.py", line 235, in check_partial
    self.expr_checker.accept(node)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checkexpr.py", line 2213, in accept
    typ = node.accept(self)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/nodes.py", line 1556, in accept
    return visitor.visit_lambda_expr(self)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checkexpr.py", line 1941, in visit_lambda_expr
    ret_type = self.accept(e.expr(), allow_none_return=True)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checkexpr.py", line 2213, in accept
    typ = node.accept(self)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/nodes.py", line 1459, in accept
    return visitor.visit_op_expr(self)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checkexpr.py", line 1276, in visit_op_expr
    return self.check_boolean_op(e, e)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checkexpr.py", line 1522, in check_boolean_op
    right_type = self.analyze_cond_branch(right_map, e.right, left_type)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checkexpr.py", line 2179, in analyze_cond_branch
    with self.chk.binder.frame_context(can_skip=True, fall_through=0):
  File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/binder.py", line 326, in frame_context
    assert len(self.frames) > 1
AssertionError: 
```

Without `--show-stacktrace`, the error is:
```
src/rooms/minerals.py:687: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.521
src/rooms/minerals.py:687: note: please use --show-traceback to print a traceback when reporting a bug
```

related lines being:
```python
686                mineral_chosen = _.find(Object.keys(self.terminal.store),
687                                        lambda r: r != RESOURCE_ENERGY and self.terminal.store[r] >= 100)
```
Could you reveal the signature of that `find()` method?
(Update: I don't think we actually need more info, since we already have a small repro [here](https://github.com/python/mypy/issues/3672#issuecomment-313802835).)
@daboross I am guessing that `self.terminal.store` or `self.terminal` is being defined after that find/lambda expression? Otherwise this may be more complicated than I thought.
@ethanhs It should all be defined before, but it might be a bit convoluted - I'm still fairly new to getting everything in typing right.

`self.terminal` is defined in `__init__`, copied from a `terminal` property of other object, which is typed as `Optional[StructureTerminal]`. `StructureTerminal.store` is a `Dict[str, int]`

other things:
```python
# Object.keys:
_K = TypeVar('_K')
_V = TypeVar('_V')
    @classmethod
    def keys(cls, obj: Dict[_K, _V]) -> List[_K]:
        pass

# _.find
_L1 = TypeVar('_L1')
_LodashPredicate = Union[Dict[str, Any], Callable[[_L1], bool], None]
_Collection = Union[List[_L1], Dict[str, _L1]]
    @staticmethod
    def find(collection: _Collection, predicate: _LodashPredicate = None, thisArg: Any = None) -> _L1:
        pass
```

I haven't gotten a successful reproduction with any more minimal cases than my entire codebase though - not sure how helpful this is.
Ok. I think for now I can work on a patch based on my repro, and you can test it once I have made the patch.
@ethanhs Are you still working on this? It looks like a user got hit by this at Dropbox.
@JukkaL I was planning on working on it when I had time this weekend. If you want the fix to make it into 0.530, feel free to go ahead and fix it, I don't think I will have time today.
This is still happening a lot for our internal codebase.
I'm currently working on my PEP implementation and spec at the moment, so perhaps someone else should take this one over?
No worries.
I can take this.
Simplified example that still causes a crash:

```py
from typing import Callable

class C:
    def f(self) -> None:
        g: Callable[[], int] = lambda: 1 or self.x
        self.x = int()
```

Resolving patch:
diff --git a/mypy/checker.py b/mypy/checker.py
--- a/mypy/checker.py
+++ b/mypy/checker.py
@@ -260,9 +260,12 @@ def check_top_level(self, node: MypyFile) -> None:
         # TODO: Handle __all__
 
     def handle_cannot_determine_type(self, name: str, context: Context) -> None:
-        node = self.scope.top_function()
-        if self.pass_num < LAST_PASS and isinstance(node, (FuncDef, LambdaExpr)):
-            # Don't report an error yet. Just defer.
+        node = self.scope.top_non_lambda_function()
+        if self.pass_num < LAST_PASS and isinstance(node, FuncDef):
+            # Don't report an error yet. Just defer. Note that we don't defer
+            # lambdas because they are coupled to the surrounding function
+            # through the binder and the inferred type of the lambda, so it
+            # would get messy.
             if self.errors.type_name:
                 type_name = self.errors.type_name[-1]
             else:
@@ -3444,6 +3447,12 @@ def top_function(self) -> Optional[FuncItem]:
                 return e
         return None
 
+    def top_non_lambda_function(self) -> Optional[FuncItem]:
+        for e in reversed(self.stack):
+            if isinstance(e, FuncItem) and not isinstance(e, LambdaExpr):
+                return e
+        return None
+
     def active_class(self) -> Optional[TypeInfo]:
         if isinstance(self.stack[-1], TypeInfo):
             return self.stack[-1]


Test patch:
diff --git a/test-data/unit/check-inference-context.test b/test-data/unit/check-inference-context.test
--- a/test-data/unit/check-inference-context.test
+++ b/test-data/unit/check-inference-context.test
@@ -911,3 +911,11 @@ class M(Generic[_KT, _VT]):
 
 def f(d: M[_KT, _VT], k: _KT) -> Union[_VT, None]:
     return d.get(k, None)
+
+[case testLambdaDeferredCrash]
+from typing import Callable
+
+class C:
+    def f(self) -> None:
+        g: Callable[[], int] = lambda: 1 or self.x
+        self.x = int()
diff --git a/test-data/unit/check-optional.test b/test-data/unit/check-optional.test
--- a/test-data/unit/check-optional.test
+++ b/test-data/unit/check-optional.test
@@ -650,3 +650,26 @@ y: Optional[int]
 x = y
 reveal_type(x) # E: Revealed type is 'Union[builtins.int, builtins.None]'
 [out]
+
+[case testNarrowOptionalOutsideLambda]
+from typing import Optional
+
+class A:
+    a: int
+
+def f(x: Optional[A]) -> None:
+    assert x
+    lambda: x.a
+[builtins fixtures/isinstancelist.pyi]
+
+[case testNarrowOptionalOutsideLambdaWithDeferred]
+from typing import Optional
+
+class A:
+    a: int
+
+    def f(self, x: Optional['A']) -> None:
+        assert x
+        lambda: (self.y, x.a) # E: Cannot determine type of 'y'
+        self.y = int()
+[builtins fixtures/isinstancelist.pyi]