Repository: python/mypy
Pull request: Last strict optional fixes
Fixes #1955 

This PR will make mypy completely ``--strict-optional`` clean. In addition to signature fixes and other minor fixes (mostly in ``semanal.py``) I introduce a change in mypy as a typechecker, I have noticed a pattern that appears several times, but was not understood by mypy:
```python
x: Optional[int]
if x in [1, 2, 3]:
    reveal_type(x)  # now this will be 'int'
```
I allow this only for built-in containers such as ``list``, ``dict``, ``set``, ``frozenset``, and ``tuple``, since other types can change the semantics of ``__contains__``. (Also this way it is a small change, a general treatment of ``Container`` subclasses will require code churn to get ``named_type``).

**UPDATE:** The feature part now has a separate PR #4072.

Problem statement:
Get mypy itself type-checking cleanly under --strict-optional
One necessary milestone toward making `--strict-optional` a recommended thing to use; the default; and ultimately the only mode (following what PEP 484 already says) will be for the mypy implementation itself to pass type-checking under `--strict-optional`. This is a tracking task for that goal.

Hints from issue discussion:
As of a version from master yesterday (commit `67d89b9e7` from 2016-07-27), this produced 480 error messages (counted with `mypy -p mypy --strict-optional 2>&1 | grep -cP '^\S*:\d+: '`) and 728 total lines of output. So there's some work to do.

I went through these errors to try to sort out which ones are bugs in mypy _in its role as the type-checker_, either in the `--strict-optional` code or exposed here by `--strict-optional`, and which ones are bugs in mypy as a statically-typed Python codebase.

Detailed notes here:
https://paper.dropbox.com/doc/Mypy-strict-optional-issues-UQPcKDn5VDdSXQme37o1G
and data here:
https://gist.github.com/gnprice/6d0413f022ca2a17d3d2bb1d4b96fbae

Short summary:
- 9 issues in mypy as the type-checker cause errors, totalling 144 lines. Of these, at least 5 are easy to work around in the target codebase; the other 4 cause 139 lines of error messages.
- 4 issues make various error messages less helpful than our usual standard of error-message quality, which could slow down fixing the `None`/`Optional`-related type errors in a codebase.
- 1 issue in the `subprocess` stub causes 2 messages.
- 334 lines of errors appear to be genuine issues in mypy as the target codebase.

I'll locate or file individual issues for the bugs in mypy-as-type-checker.

I think a lot of the 334 lines of issues in mypy-as-target-codebase will be straightforward to fix, many of them in large swaths, by adding `Optional` in appropriate places and `is None` checks in others. These often have the effect that adding `Optional` in one place causes a new error to show up in another place, so it can be hard to track progress. Probably we should fix the bugs in mypy-as-type-checker first, though, and work around any we don't yet fix, before making a big push to sort out the types.

Two of the issues are
- #1734 -- one error, easy to work around locally
- #1820 -- 13 errors, would be much better to fix

I think the other issues are as yet unfiled -- I'll start filing them.

Filed also #1956, #1957, and #1958.

The three issues that cause errors in many places are #1956, #1957, #1820.

There are four more issues (in addition to #1734 and #1958) that cause an error in just one place each and probably aren't hard to just work around where they occur. Should definitely be filed and fixed, but I'm going AFK just now -- so as a temporary breadcrumb, here they are by the shorthand titles I gave them [in my notes](https://paper.dropbox.com/doc/Mypy-strict-optional-issues-UQPcKDn5VDdSXQme37o1G):
- Tuple and empty container
- type update due to assignment
- subtype on Callable neglecting optionality of args
- list slices should be effectively covariant

FYI, for counting error messages, instead of piping mypy's output through grep you can use `--suppress-error-context`.

Commit message: Fix typeshed commit

Patch to review:
diff --git a/mypy/checker.py b/mypy/checker.py
index 35d357f8cb6f..36343c29f845 100644
--- a/mypy/checker.py
+++ b/mypy/checker.py
@@ -1814,9 +1814,9 @@ def check_multi_assignment_from_iterable(self, lvalues: List[Lvalue], rvalue_typ
     def check_lvalue(self, lvalue: Lvalue) -> Tuple[Optional[Type],
                                                     Optional[IndexExpr],
                                                     Optional[Var]]:
-        lvalue_type = None  # type: Optional[Type]
-        index_lvalue = None  # type: Optional[IndexExpr]
-        inferred = None  # type: Optional[Var]
+        lvalue_type = None
+        index_lvalue = None
+        inferred = None
 
         if self.is_definition(lvalue):
             if isinstance(lvalue, NameExpr):
@@ -2781,12 +2781,10 @@ def iterable_item_type(self, instance: Instance) -> Type:
     def function_type(self, func: FuncBase) -> FunctionLike:
         return function_type(func, self.named_type('builtins.function'))
 
-    # TODO: These next two functions should refer to TypeMap below
-    def find_isinstance_check(self, n: Expression) -> Tuple[Optional[Dict[Expression, Type]],
-                                                            Optional[Dict[Expression, Type]]]:
+    def find_isinstance_check(self, n: Expression) -> 'Tuple[TypeMap, TypeMap]':
         return find_isinstance_check(n, self.type_map)
 
-    def push_type_map(self, type_map: Optional[Dict[Expression, Type]]) -> None:
+    def push_type_map(self, type_map: 'TypeMap') -> None:
         if type_map is None:
             self.binder.unreachable()
         else:
@@ -2953,6 +2951,27 @@ def remove_optional(typ: Type) -> Type:
         return typ
 
 
+def builtin_item_type(tp: Type) -> Optional[Type]:
+    # This is only OK for built-in containers, where we know the behavior of __contains__.
+    if isinstance(tp, Instance):
+        if tp.type.fullname() in ['builtins.list', 'builtins.tuple', 'builtins.dict',
+                                  'builtins.set', 'builtins.frozenset']:
+            if not tp.args:
+                # TODO: fix tuple in lib-stub/builtins.pyi (it should be generic).
+                return None
+            if not isinstance(tp.args[0], AnyType):
+                return tp.args[0]
+    elif isinstance(tp, TupleType) and all(not isinstance(it, AnyType) for it in tp.items):
+        return UnionType.make_simplified_union(tp.items)  # this type is not externally visible
+    elif isinstance(tp, TypedDictType):
+        # TypedDict always has non-optional string keys.
+        if tp.fallback.type.fullname() == 'typing.Mapping':
+            return tp.fallback.args[0]
+        elif tp.fallback.type.bases[0].type.fullname() == 'typing.Mapping':
+            return tp.fallback.type.bases[0].args[0]
+    return None
+
+
 def and_conditional_maps(m1: TypeMap, m2: TypeMap) -> TypeMap:
     """Calculate what information we can learn from the truth of (e1 and e2)
     in terms of the information that we can learn from the truth of e1 and
@@ -3099,6 +3118,20 @@ def find_isinstance_check(node: Expression,
                     optional_expr = node.operands[1]
                 if is_overlapping_types(optional_type, comp_type):
                     return {optional_expr: remove_optional(optional_type)}, {}
+        elif node.operators in [['in'], ['not in']]:
+            expr = node.operands[0]
+            left_type = type_map[expr]
+            right_type = builtin_item_type(type_map[node.operands[1]])
+            right_ok = right_type and (not is_optional(right_type) and
+                                       (not isinstance(right_type, Instance) or
+                                        right_type.type.fullname() != 'builtins.object'))
+            if (right_type and right_ok and is_optional(left_type) and
+                    literal(expr) == LITERAL_TYPE and not is_literal_none(expr) and
+                    is_overlapping_types(left_type, right_type)):
+                if node.operators == ['in']:
+                    return {expr: remove_optional(left_type)}, {}
+                if node.operators == ['not in']:
+                    return {}, {expr: remove_optional(left_type)}
     elif isinstance(node, RefExpr):
         # Restrict the type of the variable to True-ish/False-ish in the if and else branches
         # respectively
diff --git a/mypy/checkexpr.py b/mypy/checkexpr.py
index 9098891dd559..b63400150064 100644
--- a/mypy/checkexpr.py
+++ b/mypy/checkexpr.py
@@ -721,8 +721,7 @@ def infer_arg_types_in_context2(
 
         Returns the inferred types of *actual arguments*.
         """
-        dummy = None  # type: Any
-        res = [dummy] * len(args)  # type: List[Type]
+        res = [None] * len(args)  # type: List[Optional[Type]]
 
         for i, actuals in enumerate(formal_to_actual):
             for ai in actuals:
@@ -733,7 +732,8 @@ def infer_arg_types_in_context2(
         for i, t in enumerate(res):
             if not t:
                 res[i] = self.accept(args[i])
-        return res
+        assert all(tp is not None for tp in res)
+        return cast(List[Type], res)
 
     def infer_function_type_arguments_using_context(
             self, callable: CallableType, error_context: Context) -> CallableType:
@@ -2648,7 +2648,7 @@ def is_async_def(t: Type) -> bool:
 def map_actuals_to_formals(caller_kinds: List[int],
                            caller_names: Optional[Sequence[Optional[str]]],
                            callee_kinds: List[int],
-                           callee_names: List[Optional[str]],
+                           callee_names: Sequence[Optional[str]],
                            caller_arg_type: Callable[[int],
                                                      Type]) -> List[List[int]]:
     """Calculate mapping between actual (caller) args and formals.
diff --git a/mypy/nodes.py b/mypy/nodes.py
index 761f93620de1..869f1dbf5117 100644
--- a/mypy/nodes.py
+++ b/mypy/nodes.py
@@ -4,7 +4,7 @@
 from abc import abstractmethod
 from collections import OrderedDict
 from typing import (
-    Any, TypeVar, List, Tuple, cast, Set, Dict, Union, Optional, Callable,
+    Any, TypeVar, List, Tuple, cast, Set, Dict, Union, Optional, Callable, Sequence,
 )
 
 import mypy.strconv
@@ -1786,11 +1786,11 @@ class NewTypeExpr(Expression):
     """NewType expression NewType(...)."""
     name = None  # type: str
     # The base type (the second argument to NewType)
-    old_type = None  # type: mypy.types.Type
+    old_type = None  # type: Optional[mypy.types.Type]
     # The synthesized class representing the new type (inherits old_type)
     info = None  # type: Optional[TypeInfo]
 
-    def __init__(self, name: str, old_type: 'mypy.types.Type', line: int) -> None:
+    def __init__(self, name: str, old_type: 'Optional[mypy.types.Type]', line: int) -> None:
         self.name = name
         self.old_type = old_type
 
@@ -2552,7 +2552,7 @@ def check_arg_kinds(arg_kinds: List[int], nodes: List[T], fail: Callable[[str, T
             is_kw_arg = True
 
 
-def check_arg_names(names: List[Optional[str]], nodes: List[T], fail: Callable[[str, T], None],
+def check_arg_names(names: Sequence[Optional[str]], nodes: List[T], fail: Callable[[str, T], None],
                     description: str = 'function definition') -> None:
     seen_names = set()  # type: Set[Optional[str]]
     for name, node in zip(names, nodes):
diff --git a/mypy/semanal.py b/mypy/semanal.py
index 369a30fbf67f..3450c5b8a673 100644
--- a/mypy/semanal.py
+++ b/mypy/semanal.py
@@ -192,13 +192,13 @@ class SemanticAnalyzerPass2(NodeVisitor[None]):
     # Names declated using "nonlocal" (separate set for each scope)
     nonlocal_decls = None  # type: List[Set[str]]
     # Local names of function scopes; None for non-function scopes.
-    locals = None  # type: List[SymbolTable]
+    locals = None  # type: List[Optional[SymbolTable]]
     # Nested block depths of scopes
     block_depth = None  # type: List[int]
     # TypeInfo of directly enclosing class (or None)
     type = None  # type: Optional[TypeInfo]
     # Stack of outer classes (the second tuple item contains tvars).
-    type_stack = None  # type: List[TypeInfo]
+    type_stack = None  # type: List[Optional[TypeInfo]]
     # Type variables that are bound by the directly enclosing class
     bound_tvars = None  # type: List[SymbolTableNode]
     # Type variables bound by the current scope, be it class or function
@@ -277,6 +277,7 @@ def visit_file(self, file_node: MypyFile, fnam: str, options: Options,
             for name in implicit_module_attrs:
                 v = self.globals[name].node
                 if isinstance(v, Var):
+                    assert v.type is not None, "Type of implicit attribute not set"
                     v.type = self.anal_type(v.type)
                     v.is_ready = True
 
@@ -369,6 +370,7 @@ def visit_func_def(self, defn: FuncDef) -> None:
             #   be a win.
             if self.is_class_scope():
                 # Method definition
+                assert self.type is not None, "Type not set at class scope"
                 defn.info = self.type
                 if not defn.is_decorated and not defn.is_overload:
                     if (defn.name() in self.type.names and
@@ -381,6 +383,7 @@ def visit_func_def(self, defn: FuncDef) -> None:
                 self.prepare_method_signature(defn)
             elif self.is_func_scope():
                 # Nested function
+                assert self.locals[-1] is not None, "No locals at function scope"
                 if not defn.is_decorated and not defn.is_overload:
                     if defn.name() in self.locals[-1]:
                         # Redefinition. Conditional redefinition is okay.
@@ -393,6 +396,7 @@ def visit_func_def(self, defn: FuncDef) -> None:
                 # Top-level function
                 if not defn.is_decorated and not defn.is_overload:
                     symbol = self.globals.get(defn.name())
+                    assert symbol, "Global function not found in globals"
                     if isinstance(symbol.node, FuncDef) and symbol.node != defn:
                         # This is redefinition. Conditional redefinition is okay.
                         if not self.set_original_def(symbol.node, defn):
@@ -413,14 +417,17 @@ def visit_func_def(self, defn: FuncDef) -> None:
                 else:
                     # A coroutine defined as `async def foo(...) -> T: ...`
                     # has external return type `Awaitable[T]`.
-                    defn.type = defn.type.copy_modified(
-                        ret_type = self.named_type_or_none('typing.Awaitable',
-                                                           [defn.type.ret_type]))
+                    ret_type = self.named_type_or_none('typing.Awaitable', [defn.type.ret_type])
+                    if ret_type is None:
+                        # We are running tests.
+                        ret_type = self.named_type('typing_full.Awaitable', [defn.type.ret_type])
+                    defn.type = defn.type.copy_modified(ret_type=ret_type)
             self.errors.pop_function()
 
     def prepare_method_signature(self, func: FuncDef) -> None:
         """Check basic signature validity and tweak annotation of self/cls argument."""
         # Only non-static methods are special.
+        assert self.type is not None, "This method should be only called inside a class"
         functype = func.type
         if not func.is_static:
             if not func.arguments:
@@ -434,7 +441,7 @@ def prepare_method_signature(self, func: FuncDef) -> None:
                         leading_type = fill_typevars(self.type)
                     func.type = replace_implicit_first_type(functype, leading_type)
 
-    def set_original_def(self, previous: Node, new: FuncDef) -> bool:
+    def set_original_def(self, previous: Optional[Node], new: FuncDef) -> bool:
         """If 'new' conditionally redefine 'previous', set 'previous' as original
 
         We reject straight redefinitions of functions, as they are usually
@@ -534,7 +541,7 @@ def visit_overloaded_func_def(self, defn: OverloadedFuncDef) -> None:
                 assert defn.impl is defn.items[-1]
                 defn.items = defn.items[:-1]
             elif not self.is_stub_file and not non_overload_indexes:
-                if not (self.is_class_scope() and self.type.is_protocol):
+                if not (self.type and not self.is_func_scope() and self.type.is_protocol):
                     self.fail(
                         "An overloaded function outside a stub file must have an implementation",
                         defn)
@@ -554,7 +561,7 @@ def visit_overloaded_func_def(self, defn: OverloadedFuncDef) -> None:
             # redfinitions already.
             return
 
-        if self.is_class_scope():
+        if self.type and not self.is_func_scope():
             self.type.names[defn.name()] = SymbolTableNode(MDEF, defn,
                                                            typ=defn.type)
             defn.info = self.type
@@ -629,9 +636,8 @@ def analyze_function(self, defn: FuncItem) -> None:
             self.function_stack.pop()
 
     def check_classvar_in_signature(self, typ: Type) -> None:
-        t = None  # type: Type
         if isinstance(typ, Overloaded):
-            for t in typ.items():
+            for t in typ.items():  # type: Type
                 self.check_classvar_in_signature(t)
             return
         if not isinstance(typ, CallableType):
@@ -690,8 +696,10 @@ def analyze_class_body(self, defn: ClassDef) -> Iterator[bool]:
                     if prohibited in named_tuple_info.names:
                         if nt_names.get(prohibited) is named_tuple_info.names[prohibited]:
                             continue
+                        ctx = named_tuple_info.names[prohibited].node
+                        assert ctx is not None
                         self.fail('Cannot overwrite NamedTuple attribute "{}"'.format(prohibited),
-                                  named_tuple_info.names[prohibited].node)
+                                  ctx)
 
                 # Restore the names in the original symbol table. This ensures that the symbol
                 # table contains the field objects created by build_namedtuple_typeinfo. Exclude
@@ -759,7 +767,7 @@ def calculate_abstract_status(self, typ: TypeInfo) -> None:
                     # check arbitrarily the first overload item. If the
                     # different items have a different abstract status, there
                     # should be an error reported elsewhere.
-                    func = node.items[0]  # type: Node
+                    func = node.items[0]  # type: Optional[Node]
                 else:
                     func = node
                 if isinstance(func, Decorator):
@@ -779,7 +787,7 @@ def setup_type_promotion(self, defn: ClassDef) -> None:
 
         This includes things like 'int' being compatible with 'float'.
         """
-        promote_target = None  # type: Type
+        promote_target = None  # type: Optional[Type]
         for decorator in defn.decorators:
             if isinstance(decorator, CallExpr):
                 analyzed = decorator.analyzed
@@ -886,14 +894,14 @@ def analyze_typevar_declaration(self, t: Type) -> Optional[TypeVarList]:
             return tvars
         return None
 
-    def analyze_unbound_tvar(self, t: Type) -> Tuple[str, TypeVarExpr]:
+    def analyze_unbound_tvar(self, t: Type) -> Optional[Tuple[str, TypeVarExpr]]:
         if not isinstance(t, UnboundType):
             return None
         unbound = t
         sym = self.lookup_qualified(unbound.name, unbound)
         if sym is None or sym.kind != TVAR:
             return None
-        elif not self.tvar_scope.allow_binding(sym.fullname):
+        elif sym.fullname and not self.tvar_scope.allow_binding(sym.fullname):
             # It's bound by our type variable scope
             return None
         else:
@@ -1138,6 +1146,7 @@ def expr_to_analyzed_type(self, expr: Expression) -> Type:
                 # Some form of namedtuple is the only valid type that looks like a call
                 # expression. This isn't a valid type.
                 raise TypeTranslationError()
+            assert info.tuple_type, "NamedTuple without tuple type"
             fallback = Instance(info, [])
             return TupleType(info.tuple_type.items, fallback=fallback)
         typ = expr_to_unanalyzed_type(expr)
@@ -1178,11 +1187,12 @@ def is_base_class(self, t: TypeInfo, s: TypeInfo) -> bool:
 
     def analyze_metaclass(self, defn: ClassDef) -> None:
         if defn.metaclass:
+            metaclass_name = None
             if isinstance(defn.metaclass, NameExpr):
                 metaclass_name = defn.metaclass.name
             elif isinstance(defn.metaclass, MemberExpr):
                 metaclass_name = get_member_expr_fullname(defn.metaclass)
-            else:
+            if metaclass_name is None:
                 self.fail("Dynamic metaclass not supported for '%s'" % defn.name, defn.metaclass)
                 return
             sym = self.lookup_qualified(metaclass_name, defn.metaclass)
@@ -1229,8 +1239,9 @@ def class_type(self, info: TypeInfo) -> Type:
         else:
             return leading_type
 
-    def named_type(self, qualified_name: str, args: List[Type] = None) -> Instance:
-        sym = self.lookup_qualified(qualified_name, None)
+    def named_type(self, qualified_name: str, args: Optional[List[Type]] = None) -> Instance:
+        sym = self.lookup_qualified(qualified_name, Context())
+        assert sym, "Internal error: attempted to construct unknown type"
         node = sym.node
         assert isinstance(node, TypeInfo)
         if args:
@@ -1238,7 +1249,8 @@ def named_type(self, qualified_name: str, args: List[Type] = None) -> Instance:
             return Instance(node, args)
         return Instance(node, [AnyType(TypeOfAny.special_form)] * len(node.defn.type_vars))
 
-    def named_type_or_none(self, qualified_name: str, args: List[Type] = None) -> Instance:
+    def named_type_or_none(self, qualified_name: str,
+                           args: Optional[List[Type]] = None) -> Optional[Instance]:
         sym = self.lookup_fully_qualified_or_none(qualified_name)
         if not sym:
             return None
@@ -1316,9 +1328,9 @@ def analyze_typeddict_classdef(self, defn: ClassDef) -> bool:
         return False
 
     def check_typeddict_classdef(self, defn: ClassDef,
-                                 oldfields: List[str] = None) -> Tuple[List[str],
-                                                                       List[Type],
-                                                                       Set[str]]:
+                                 oldfields: Optional[List[str]] = None) -> Tuple[List[str],
+                                                                                 List[Type],
+                                                                                 Set[str]]:
         TPDICT_CLASS_ERROR = ('Invalid statement in TypedDict definition; '
                               'expected "field_name: field_type"')
         if self.options.python_version < (3, 6):
@@ -1356,7 +1368,7 @@ def check_typeddict_classdef(self, defn: ClassDef,
                 elif not isinstance(stmt.rvalue, TempNode):
                     # x: int assigns rvalue to TempNode(AnyType())
                     self.fail('Right hand side values are not supported in TypedDict', stmt)
-        total = True
+        total = True  # type: Optional[bool]
         if 'total' in defn.keywords:
             total = self.parse_bool(defn.keywords['total'])
             if total is None:
@@ -1501,24 +1513,25 @@ def process_import_over_existing_name(self,
         return False
 
     def normalize_type_alias(self, node: SymbolTableNode,
-                             ctx: Context) -> SymbolTableNode:
+                             ctx: Context) -> Optional[SymbolTableNode]:
         normalized = False
         fullname = node.fullname
         if fullname in type_aliases:
             # Node refers to an aliased type such as typing.List; normalize.
-            node = self.lookup_qualified(type_aliases[fullname], ctx)
-            if node is None:
+            new_node = self.lookup_qualified(type_aliases[fullname], ctx)
+            if new_node is None:
                 self.add_fixture_note(fullname, ctx)
                 return None
             normalized = True
         if fullname in collections_type_aliases:
             # Similar, but for types from the collections module like typing.DefaultDict
             self.add_module_symbol('collections', '__mypy_collections__', False, ctx)
-            node = self.lookup_qualified(collections_type_aliases[fullname], ctx)
+            new_node = self.lookup_qualified(collections_type_aliases[fullname], ctx)
             normalized = True
         if normalized:
-            node = SymbolTableNode(node.kind, node.node, node.type_override,
-                                   normalized=True, alias_tvars=node.alias_tvars)
+            assert new_node is not None, "Collection node not found"
+            node = SymbolTableNode(new_node.kind, new_node.node, new_node.type_override,
+                                   normalized=True, alias_tvars=new_node.alias_tvars)
         return node
 
     def add_fixture_note(self, fullname: str, ctx: Context) -> None:
@@ -1551,20 +1564,21 @@ def visit_import_all(self, i: ImportAll) -> None:
             m = self.modules[i_id]
             self.add_submodules_to_parent_modules(i_id, True)
             for name, node in m.names.items():
-                node = self.normalize_type_alias(node, i)
+                new_node = self.normalize_type_alias(node, i)
                 # if '__all__' exists, all nodes not included have had module_public set to
                 # False, and we can skip checking '_' because it's been explicitly included.
-                if node.module_public and (not name.startswith('_') or '__all__' in m.names):
+                if (new_node and new_node.module_public and
+                        (not name.startswith('_') or '__all__' in m.names)):
                     existing_symbol = self.globals.get(name)
                     if existing_symbol:
                         # Import can redefine a variable. They get special treatment.
                         if self.process_import_over_existing_name(
-                                name, existing_symbol, node, i):
+                                name, existing_symbol, new_node, i):
                             continue
-                    self.add_symbol(name, SymbolTableNode(node.kind, node.node,
-                                                          node.type_override,
-                                                          normalized=node.normalized,
-                                                          alias_tvars=node.alias_tvars), i)
+                    self.add_symbol(name, SymbolTableNode(new_node.kind, new_node.node,
+                                                          new_node.type_override,
+                                                          normalized=new_node.normalized,
+                                                          alias_tvars=new_node.alias_tvars), i)
         else:
             # Don't add any dummy symbols for 'from x import *' if 'x' is unknown.
             pass
@@ -1596,7 +1610,7 @@ def visit_block(self, b: Block) -> None:
             self.accept(s)
         self.block_depth[-1] -= 1
 
-    def visit_block_maybe(self, b: Block) -> None:
+    def visit_block_maybe(self, b: Optional[Block]) -> None:
         if b:
             self.visit_block(b)
 
@@ -1628,16 +1642,11 @@ def anal_type(self, t: Type, *,
                   allow_tuple_literal: bool = False,
                   aliasing: bool = False,
                   third_pass: bool = False) -> Type:
-        if t:
-            a = self.type_analyzer(
-                tvar_scope=tvar_scope,
-                aliasing=aliasing,
-                allow_tuple_literal=allow_tuple_literal,
-                third_pass=third_pass)
-            return t.accept(a)
-
-        else:
-            return None
+        a = self.type_analyzer(tvar_scope=tvar_scope,
+                               aliasing=aliasing,
+                               allow_tuple_literal=allow_tuple_literal,
+                               third_pass=third_pass)
+        return t.accept(a)
 
     def visit_assignment_stmt(self, s: AssignmentStmt) -> None:
         for lval in s.lvalues:
@@ -1771,6 +1780,7 @@ def check_and_set_up_type_alias(self, s: AssignmentStmt) -> None:
         if not res:
             return
         node = self.lookup(lvalue.name, lvalue)
+        assert node is not None
         if not lvalue.is_def:
             # Type aliases can't be re-defined.
             if node and (node.kind == TYPE_ALIAS or isinstance(node.node, TypeInfo)):
@@ -1833,7 +1843,7 @@ def analyze_lvalue(self, lval: Lvalue, nested: bool = False,
                 # Since the is_def flag is set, this must have been analyzed
                 # already in the first pass and added to the symbol table.
                 assert lval.node.name() in self.globals
-            elif (self.is_func_scope() and lval.name not in self.locals[-1] and
+            elif (self.locals[-1] is not None and lval.name not in self.locals[-1] and
                   lval.name not in self.global_decls[-1] and
                   lval.name not in self.nonlocal_decls[-1]):
                 # Define new local name.
@@ -1908,6 +1918,7 @@ def analyze_tuple_or_list_lvalue(self, lval: Union[ListExpr, TupleExpr],
     def analyze_member_lvalue(self, lval: MemberExpr) -> None:
         lval.accept(self)
         if self.is_self_member_ref(lval):
+            assert self.type, "Self member outside a class"
             node = self.type.get(lval.name)
             if node is None or isinstance(node.node, Var) and node.node.is_abstract_var:
                 if self.type.is_protocol and node is None:
@@ -1932,7 +1943,8 @@ def is_self_member_ref(self, memberexpr: MemberExpr) -> bool:
         node = memberexpr.expr.node
         return isinstance(node, Var) and node.is_self
 
-    def check_lvalue_validity(self, node: Union[Expression, SymbolNode], ctx: Context) -> None:
+    def check_lvalue_validity(self, node: Union[Expression, SymbolNode, None],
+                              ctx: Context) -> None:
         if isinstance(node, TypeVarExpr):
             self.fail('Invalid assignment target', ctx)
         elif isinstance(node, TypeInfo):
@@ -2125,6 +2137,8 @@ def process_typevar_declaration(self, s: AssignmentStmt) -> None:
                                    context=s)
         # Yes, it's a valid type variable definition! Add it to the symbol table.
         node = self.lookup(name, s)
+        assert node is not None
+        assert node.fullname is not None
         node.kind = TVAR
         TypeVar = TypeVarExpr(name, node.fullname, values, upper_bound, variance)
         TypeVar.line = call.line
@@ -2238,10 +2252,12 @@ def process_namedtuple_definition(self, s: AssignmentStmt) -> None:
             return
         # Yes, it's a valid namedtuple definition. Add it to the symbol table.
         node = self.lookup(name, s)
+        assert node is not None
         node.kind = GDEF   # TODO locally defined namedtuple
         node.node = named_tuple
 
-    def check_namedtuple(self, node: Expression, var_name: str = None) -> Optional[TypeInfo]:
+    def check_namedtuple(self, node: Expression,
+                         var_name: Optional[str] = None) -> Optional[TypeInfo]:
         """Check if a call defines a namedtuple.
 
         The optional var_name argument is the name of the variable to
@@ -2388,7 +2404,8 @@ def build_namedtuple_typeinfo(self, name: str, items: List[str], types: List[Typ
         info.tuple_type = TupleType(types, fallback)
 
         def patch() -> None:
-            # Calculate the correct value type for the fallback Mapping.
+            # Calculate the correct value type for the fallback tuple.
+            assert info.tuple_type, "TupleType type deleted before calling the patch"
             fallback.args[0] = join.join_type_list(list(info.tuple_type.items))
 
         # We can't calculate the complete fallback type until after semantic
@@ -2421,7 +2438,7 @@ def add_field(var: Var, is_initialized_in_class: bool = False,
         def add_method(funcname: str,
                        ret: Type,
                        args: List[Argument],
-                       name: str = None,
+                       name: Optional[str] = None,
                        is_classmethod: bool = False,
                        ) -> None:
             if is_classmethod:
@@ -2433,7 +2450,9 @@ def add_method(funcname: str,
             types = [arg.type_annotation for arg in args]
             items = [arg.variable.name() for arg in args]
             arg_kinds = [arg.kind for arg in args]
-            signature = CallableType(types, arg_kinds, items, ret, function_type,
+            assert None not in types
+            signature = CallableType(cast(List[Type], types), arg_kinds, items, ret,
+                                     function_type,
                                      name=name or info.name() + '.' + funcname)
             signature.variables = [tvd]
             func = FuncDef(funcname, args, Block([]), typ=signature)
@@ -2494,7 +2513,8 @@ def process_typeddict_definition(self, s: AssignmentStmt) -> None:
             node.kind = GDEF   # TODO locally defined TypedDict
             node.node = typed_dict
 
-    def check_typeddict(self, node: Expression, var_name: str = None) -> Optional[TypeInfo]:
+    def check_typeddict(self, node: Expression,
+                        var_name: Optional[str] = None) -> Optional[TypeInfo]:
         """Check if a call defines a TypedDict.
 
         The optional var_name argument is the name of the variable to
@@ -2514,7 +2534,7 @@ def check_typeddict(self, node: Expression, var_name: str = None) -> Optional[Ty
         fullname = callee.fullname
         if fullname != 'mypy_extensions.TypedDict':
             return None
-        items, types, total, ok = self.parse_typeddict_args(call, fullname)
+        items, types, total, ok = self.parse_typeddict_args(call)
         if not ok:
             # Error. Construct dummy return value.
             info = self.build_typeddict_typeinfo('TypedDict', [], [], set())
@@ -2540,8 +2560,7 @@ def check_typeddict(self, node: Expression, var_name: str = None) -> Optional[Ty
         call.analyzed.set_line(call.line, call.column)
         return info
 
-    def parse_typeddict_args(self, call: CallExpr,
-                             fullname: str) -> Tuple[List[str], List[Type], bool, bool]:
+    def parse_typeddict_args(self, call: CallExpr) -> Tuple[List[str], List[Type], bool, bool]:
         # TODO: Share code with check_argument_count in checkexpr.py?
         args = call.args
         if len(args) < 2:
@@ -2560,7 +2579,7 @@ def parse_typeddict_args(self, call: CallExpr,
         if not isinstance(args[1], DictExpr):
             return self.fail_typeddict_arg(
                 "TypedDict() expects a dictionary literal as the second argument", call)
-        total = True
+        total = True  # type: Optional[bool]
         if len(args) == 3:
             total = self.parse_bool(call.args[2])
             if total is None:
@@ -2576,6 +2595,7 @@ def parse_typeddict_args(self, call: CallExpr,
             for t in types:
                 if has_any_from_unimported_type(t):
                     self.msg.unimported_type_becomes_any("Type of a TypedDict key", t, dictexpr)
+        assert total is not None
         return items, types, total, ok
 
     def parse_bool(self, expr: Expression) -> Optional[bool]:
@@ -2621,6 +2641,7 @@ def build_typeddict_typeinfo(self, name: str, items: List[str],
 
         def patch() -> None:
             # Calculate the correct value type for the fallback Mapping.
+            assert info.typeddict_type, "TypedDict type deleted before calling the patch"
             fallback.args[1] = join.join_type_list(list(info.typeddict_type.items.values()))
 
         # We can't calculate the complete fallback type until after semantic
@@ -2633,7 +2654,7 @@ def check_classvar(self, s: AssignmentStmt) -> None:
         lvalue = s.lvalues[0]
         if len(s.lvalues) != 1 or not isinstance(lvalue, RefExpr):
             return
-        if not self.is_classvar(s.type):
+        if not s.type or not self.is_classvar(s.type):
             return
         if self.is_class_scope() and isinstance(lvalue, NameExpr):
             node = lvalue.node
@@ -2730,7 +2751,8 @@ def process_enum_call(self, s: AssignmentStmt) -> None:
             node.kind = GDEF   # TODO locally defined Enum
             node.node = enum_call
 
-    def check_enum_call(self, node: Expression, var_name: str = None) -> Optional[TypeInfo]:
+    def check_enum_call(self, node: Expression,
+                        var_name: Optional[str] = None) -> Optional[TypeInfo]:
         """Check if a call defines an Enum.
 
         Example:
@@ -3235,7 +3257,7 @@ def translate_dict_call(self, call: CallExpr) -> Optional[DictExpr]:
             for a in call.args:
                 a.accept(self)
             return None
-        expr = DictExpr([(StrExpr(key), value)
+        expr = DictExpr([(StrExpr(cast(str, key)), value)  # since they are all ARG_NAMED
                          for key, value in zip(call.arg_names, call.args)])
         expr.set_line(call)
         expr.accept(self)
@@ -3367,6 +3389,7 @@ def visit_index_expr(self, expr: IndexExpr) -> None:
             # Special form -- subscripting a generic type alias.
             # Perform the type substitution and create a new alias.
             res, alias_tvars = self.analyze_alias(expr)
+            assert res is not None, "Failed analyzing already defined alias"
             expr.analyzed = TypeAliasExpr(res, alias_tvars, fallback=self.alias_fallback(res),
                                           in_runtime=True)
             expr.analyzed.line = expr.line
@@ -3541,7 +3564,7 @@ def lookup(self, name: str, ctx: Context,
                     self.name_not_defined(name, ctx)
                 return None
         # 2. Class attributes (if within class definition)
-        if self.is_class_scope() and name in self.type.names:
+        if self.type and not self.is_func_scope() and name in self.type.names:
             node = self.type.names[name]
             if not node.implicit:
                 return node
@@ -3641,13 +3664,13 @@ def lookup_fully_qualified(self, name: str) -> SymbolTableNode:
             next_sym = n.names[parts[i]]
             assert isinstance(next_sym.node, MypyFile)
             n = next_sym.node
-        return n.names.get(parts[-1])
+        return n.names[parts[-1]]
 
     def lookup_fully_qualified_or_none(self, name: str) -> Optional[SymbolTableNode]:
         """Lookup a fully qualified name.
 
-        Assume that the name is defined. This happens in the global namespace -- the local
-        module namespace is ignored.
+        Don't assume that the name is defined. This happens in the global namespace --
+        the local module namespace is ignored.
         """
         assert '.' in name
         parts = name.split('.')
@@ -3692,6 +3715,7 @@ def is_module_scope(self) -> bool:
     def add_symbol(self, name: str, node: SymbolTableNode,
                    context: Context) -> None:
         if self.is_func_scope():
+            assert self.locals[-1] is not None
             if name in self.locals[-1]:
                 # Flag redefinition unless this is a reimport of a module.
                 if not (node.kind == MODULE_REF and
@@ -3715,6 +3739,7 @@ def add_symbol(self, name: str, node: SymbolTableNode,
             self.globals[name] = node
 
     def add_local(self, node: Union[Var, FuncDef, OverloadedFuncDef], ctx: Context) -> None:
+        assert self.locals[-1] is not None, "Should not add locals outside a function"
         name = node.name()
         if name in self.locals[-1]:
             self.name_already_defined(name, ctx)
@@ -3838,7 +3863,7 @@ def calculate_class_mro(defn: ClassDef, fail: Callable[[str, Context], None]) ->
     defn.info.fallback_to_any = any(baseinfo.fallback_to_any for baseinfo in defn.info.mro)
 
 
-def find_duplicate(list: List[T]) -> T:
+def find_duplicate(list: List[T]) -> Optional[T]:
     """If the list has duplicates, return one of the duplicates.
 
     Otherwise, return None.
diff --git a/mypy/semanal_pass3.py b/mypy/semanal_pass3.py
index 9e9ba0d0deff..859a6f033dcc 100644
--- a/mypy/semanal_pass3.py
+++ b/mypy/semanal_pass3.py
@@ -152,7 +152,7 @@ def visit_decorator(self, dec: Decorator) -> None:
         for expr in dec.decorators:
             preserve_type = False
             if isinstance(expr, RefExpr) and isinstance(expr.node, FuncDef):
-                if is_identity_signature(expr.node.type):
+                if expr.node.type and is_identity_signature(expr.node.type):
                     preserve_type = True
             if not preserve_type:
                 decorator_preserves_type = False
@@ -246,16 +246,20 @@ def perform_transform(self, node: Union[Node, SymbolTableNode],
                           transform: Callable[[Type], Type]) -> None:
         """Apply transform to all types associated with node."""
         if isinstance(node, ForStmt):
-            node.index_type = transform(node.index_type)
+            if node.index_type:
+                node.index_type = transform(node.index_type)
             self.transform_types_in_lvalue(node.index, transform)
         if isinstance(node, WithStmt):
-            node.target_type = transform(node.target_type)
+            if node.target_type:
+                node.target_type = transform(node.target_type)
             for n in node.target:
                 if isinstance(n, NameExpr) and isinstance(n.node, Var) and n.node.type:
                     n.node.type = transform(n.node.type)
         if isinstance(node, (FuncDef, CastExpr, AssignmentStmt, TypeAliasExpr, Var)):
+            assert node.type, "Scheduled patch for non-existent type"
             node.type = transform(node.type)
         if isinstance(node, NewTypeExpr):
+            assert node.old_type, "Scheduled patch for non-existent type"
             node.old_type = transform(node.old_type)
         if isinstance(node, TypeVarExpr):
             if node.upper_bound:
@@ -263,14 +267,17 @@ def perform_transform(self, node: Union[Node, SymbolTableNode],
             if node.values:
                 node.values = [transform(v) for v in node.values]
         if isinstance(node, TypedDictExpr):
+            assert node.info.typeddict_type, "Scheduled patch for non-existent type"
             node.info.typeddict_type = cast(TypedDictType,
                                             transform(node.info.typeddict_type))
         if isinstance(node, NamedTupleExpr):
+            assert node.info.tuple_type, "Scheduled patch for non-existent type"
             node.info.tuple_type = cast(TupleType,
                                         transform(node.info.tuple_type))
         if isinstance(node, TypeApplication):
             node.types = [transform(t) for t in node.types]
         if isinstance(node, SymbolTableNode):
+            assert node.type_override, "Scheduled patch for non-existent type"
             node.type_override = transform(node.type_override)
         if isinstance(node, TypeInfo):
             for tvar in node.defn.type_vars:
@@ -296,7 +303,8 @@ def transform_types_in_lvalue(self, lvalue: Lvalue,
         if isinstance(lvalue, RefExpr):
             if isinstance(lvalue.node, Var):
                 var = lvalue.node
-                var.type = transform(var.type)
+                if var.type:
+                    var.type = transform(var.type)
         elif isinstance(lvalue, TupleExpr):
             for item in lvalue.items:
                 self.transform_types_in_lvalue(item, transform)
@@ -355,7 +363,7 @@ def fail(self, msg: str, ctx: Context, *, blocker: bool = False) -> None:
     def fail_blocker(self, msg: str, ctx: Context) -> None:
         self.fail(msg, ctx, blocker=True)
 
-    def builtin_type(self, name: str, args: List[Type] = None) -> Instance:
+    def builtin_type(self, name: str, args: Optional[List[Type]] = None) -> Instance:
         names = self.modules['builtins']
         sym = names.names[name]
         node = sym.node
diff --git a/mypy/typeanal.py b/mypy/typeanal.py
index 6de9b1e1dd95..d45a777d9853 100644
--- a/mypy/typeanal.py
+++ b/mypy/typeanal.py
@@ -53,7 +53,7 @@
 
 
 def analyze_type_alias(node: Expression,
-                       lookup_func: Callable[[str, Context], SymbolTableNode],
+                       lookup_func: Callable[[str, Context], Optional[SymbolTableNode]],
                        lookup_fqn_func: Callable[[str], SymbolTableNode],
                        tvar_scope: TypeVarScope,
                        fail_func: Callable[[str, Context], None],
@@ -144,7 +144,7 @@ class TypeAnalyser(SyntheticTypeVisitor[Type], AnalyzerPluginInterface):
     global_scope = True  # type: bool
 
     def __init__(self,
-                 lookup_func: Callable[[str, Context], SymbolTableNode],
+                 lookup_func: Callable[[str, Context], Optional[SymbolTableNode]],
                  lookup_fqn_func: Callable[[str], SymbolTableNode],
                  tvar_scope: Optional[TypeVarScope],
                  fail_func: Callable[[str, Context], None],
@@ -555,7 +555,9 @@ def bind_function_type_variables(self,
             return []  # We are in third pass, nothing new here
         if fun_type.variables:
             for var in fun_type.variables:
-                var_expr = self.lookup(var.name, var).node
+                var_node = self.lookup(var.name, var)
+                assert var_node, "Binding for function type variable not found within function"
+                var_expr = var_node.node
                 assert isinstance(var_expr, TypeVarExpr)
                 self.tvar_scope.bind(var.name, var_expr)
             return fun_type.variables
@@ -575,8 +577,12 @@ def bind_function_type_variables(self,
         return defs
 
     def is_defined_type_var(self, tvar: str, context: Context) -> bool:
-        return (self.tvar_scope is not None and
-                self.tvar_scope.get_binding(self.lookup(tvar, context)) is not None)
+        if self.tvar_scope is None:
+            return False
+        tvar_node = self.lookup(tvar, context)
+        if not tvar_node:
+            return False
+        return self.tvar_scope.get_binding(tvar_node) is not None
 
     def anal_array(self, a: List[Type], nested: bool = True) -> List[Type]:
         res = []  # type: List[Type]
@@ -638,7 +644,7 @@ class TypeAnalyserPass3(TypeVisitor[None]):
     """
 
     def __init__(self,
-                 lookup_func: Callable[[str, Context], SymbolTableNode],
+                 lookup_func: Callable[[str, Context], Optional[SymbolTableNode]],
                  lookup_fqn_func: Callable[[str], SymbolTableNode],
                  fail_func: Callable[[str, Context], None],
                  note_func: Callable[[str, Context], None],
@@ -878,7 +884,7 @@ def flatten_tvars(ll: Iterable[List[T]]) -> List[T]:
 class TypeVariableQuery(TypeQuery[TypeVarList]):
 
     def __init__(self,
-                 lookup: Callable[[str, Context], SymbolTableNode],
+                 lookup: Callable[[str, Context], Optional[SymbolTableNode]],
                  scope: 'TypeVarScope',
                  *,
                  include_callables: bool = True,
diff --git a/mypy/types.py b/mypy/types.py
index 67efd22da649..ec2a0594e666 100644
--- a/mypy/types.py
+++ b/mypy/types.py
@@ -5,7 +5,7 @@
 from collections import OrderedDict
 from typing import (
     Any, TypeVar, Dict, List, Tuple, cast, Generic, Set, Optional, Union, Iterable, NamedTuple,
-    Callable
+    Callable, Sequence
 )
 
 import mypy.nodes
@@ -642,7 +642,7 @@ class CallableType(FunctionLike):
     def __init__(self,
                  arg_types: List[Type],
                  arg_kinds: List[int],
-                 arg_names: List[Optional[str]],
+                 arg_names: Sequence[Optional[str]],
                  ret_type: Type,
                  fallback: Instance,
                  name: Optional[str] = None,
@@ -663,7 +663,7 @@ def __init__(self,
         assert not any(tp is None for tp in arg_types), "No annotation must be Any, not None"
         self.arg_types = arg_types
         self.arg_kinds = arg_kinds
-        self.arg_names = arg_names
+        self.arg_names = list(arg_names)
         self.min_args = arg_kinds.count(ARG_POS)
         self.is_var_arg = ARG_STAR in arg_kinds
         self.is_kw_arg = ARG_STAR2 in arg_kinds
diff --git a/mypy_self_check.ini b/mypy_self_check.ini
index bb30adf177e7..610e3d46f4b3 100644
--- a/mypy_self_check.ini
+++ b/mypy_self_check.ini
@@ -8,12 +8,3 @@ disallow_any = generics, unimported
 warn_redundant_casts = True
 warn_unused_ignores = True
 
-# historical exception
-[mypy-mypy.semanal]
-strict_optional = False
-
-[mypy-mypy.semanal_pass1]
-strict_optional = False
-
-[mypy-mypy.semanal_pass3]
-strict_optional = False


Code review comments:
diff_hunk: @@ -393,6 +396,7 @@ def visit_func_def(self, defn: FuncDef) -> None:
                 # Top-level function
                 if not defn.is_decorated and not defn.is_overload:
                     symbol = self.globals.get(defn.name())
+                    assert symbol, "Global function not found in globals"
path: mypy/semanal.py
text: @user1:
We could instead just use `self.globals[defn.name()]` above?

diff_hunk: @@ -413,14 +417,17 @@ def visit_func_def(self, defn: FuncDef) -> None:
                 else:
                     # A coroutine defined as `async def foo(...) -> T: ...`
                     # has external return type `Awaitable[T]`.
-                    defn.type = defn.type.copy_modified(
-                        ret_type = self.named_type_or_none('typing.Awaitable',
-                                                           [defn.type.ret_type]))
+                    ret_type = self.named_type_or_none('typing.Awaitable', [defn.type.ret_type])
+                    if ret_type is None:
+                        # We are running tests.
+                        ret_type = self.named_type('typing_full.Awaitable', [defn.type.ret_type])
+                    defn.type = defn.type.copy_modified(ret_type=ret_type)
             self.errors.pop_function()
 
     def prepare_method_signature(self, func: FuncDef) -> None:
         """Check basic signature validity and tweak annotation of self/cls argument."""
         # Only non-static methods are special.
+        assert self.type is not None, "This method should be only called inside a class"
path: mypy/semanal.py
text: @user1:
You could remove this assert by adding a `TypeInfo` argument and passing it from the caller.