Repository: python/mypy
Pull request: Allow ellipsis default args in "trivial" non-stub functions
Resolves https://github.com/python/mypy/issues/6531

This diff modifies the checker to allow default arguments to be used in function definitions in non-stub files as long as the body of that function is either empty or contains just a `raise NotImplementedError`. (This is after ignoring docstrings.)

For rationale on why this is useful, see the issue linked above.

One note on function bodies containing only a `raise Blah`. I was worried that letting mypy ignore the ellipsis when the function raises *any* Exception would lead to unsafe behavior -- I didn't want mypy to start allowing code that looks like this:

    def halt(self, reason: str = ...) -> NoReturn:
        raise MyCustomError("Fatal error: " + reason, self.line, self.current_context)

Whitelisting only NotImplementedError seemed like a good enough compromise -- it's unlikely users will use the arguments in non-trivial ways when raising that particular exception. It should also be easy enough to relax this restriction in the future if necessary.

Problem statement:
Support ellipsis default arguments in runtime functions missing a body
Mypy (and PEP 484) currently allows using ellipsis as the default argument for functions defined inside stubs:

    # Legal inside *.pyi files, but illegal inside *.py files -- we get the following error:
    # "Incompatible default for argument "some_arg" (default has type "ellipsis", argument has type "int")"
    def foo(some_arg: int = ...) -> None: pass

I propose generalizing the way mypy handles ellipsis default arguments so that the above becomes permitted for *both* stub files and regular files as long as the body of the function is either empty or contains just a `raise SomeException(...)` statement.

The main reason I want this is because I was trying to define a custom Callback protocol where one of the arguments accepted a mutable argument, like so:

    class MyCallback(Protocol):
        def __call__(self, foo: List[int] = ???) -> None: ...

Unfortunately, there doesn't seem to be an entirely clean way of supporting this:

1. I could do `foo: List[int] = []`, but then I get warnings from my IDE and linter about having a mutable default argument.
2. I could do `foo: Optional[List[int]] = None`, but I don't actually want to allow `None` arguments.
3. I could do `foo: List[int] = bogus`, where `bogus` has type `Any`, but that causes another error when using the `--disallow-any-explicit` flag.

You run into similar issues when trying to define overloads variants or abstract methods -- basically, function definitions that effectively exist only at "type-check time". 

The restriction that you can only use ellipsis default arguments when the function body is missing/just throws an exception should hopefully help prevent the ellipsis from appearing in unexpected places and causing runtime errors.

I'd be willing to implement this myself (especially since this seems like a pretty low-priority feature, all things considered), but want to make sure the idea seems like a reasonable one first before investing the effort.

Hints from issue discussion:
I like this idea, this can be useful for both protocols and ABCs.

Commit message: Respond to a code review I missed

Patch to review:
diff --git a/docs/source/stubs.rst b/docs/source/stubs.rst
index eb68dba35e4b..5ba67ffaeb0c 100644
--- a/docs/source/stubs.rst
+++ b/docs/source/stubs.rst
@@ -62,20 +62,81 @@ Stub file syntax
 
 Stub files are written in normal Python 3 syntax, but generally
 leaving out runtime logic like variable initializers, function bodies,
-and default arguments, or replacing them with ellipses.
+and default arguments.
 
-In this example, each ellipsis ``...`` is literally written in the
+If it is not possible to completely leave out some piece of runtime
+logic, the recommended convention is to replace or elide them with ellipsis
+expressions (``...``). Each ellipsis below is literally written in the
 stub file as three dots:
 
 .. code-block:: python
 
+    # Variables with annotations do not need to be assigned a value.
+    # So by convention, we omit them in the stub file.
     x: int
 
-    def afunc(code: str) -> int: ...
-    def afunc(a: int, b: int = ...) -> int: ...
+    # Function bodies cannot be completely removed. By convention,
+    # we replace them with `...` instead of the `pass` statement.
+    def func_1(code: str) -> int: ...
+
+    # We can do the same with default arguments.
+    def func_2(a: int, b: int = ...) -> int: ...
 
 .. note::
 
     The ellipsis ``...`` is also used with a different meaning in
     :ref:`callable types <callable-types>` and :ref:`tuple types
     <tuple-types>`.
+
+.. note::
+
+    It is always legal to use Python 3 syntax in stub files, even when
+    writing Python 2 code. The example above is a valid stub file
+    for both Python 2 and 3.
+
+Using stub file syntax at runtime
+*********************************
+
+You may also occasionally need to elide actual logic in regular
+Python code -- for example, when writing methods in
+:ref:`overload variants <function-overloading>` or
+:ref:`custom protocols <protocol-types>`.
+
+The recommended style is to use ellipses to do so, just like in
+stub files. It is also considered stylistically acceptable to
+throw a ``NotImplementedError`` in cases where the user of the
+code may accidentally call functions with no actual logic.
+
+You can also elide default arguments as long as the function body
+also contains no runtime logic: the function body only contains
+a single ellipsis, the pass statement, or a ``raise NotImplementedError()``.
+It is also ok for the function body to contain a docstring.
+For example:
+
+.. code-block:: python
+
+    from typing import List
+    from typing_extensions import Protocol
+
+    class Resource(Protocol):
+        def ok_1(self, foo: List[str] = ...) -> None: ...
+
+        def ok_2(self, foo: List[str] = ...) -> None:
+            raise NotImplementedError()
+
+        def ok_3(self, foo: List[str] = ...) -> None:
+            """Some docstring"""
+            pass
+
+        # Error: Incompatible default for argument "foo" (default has
+        # type "ellipsis", argument has type "List[str]")
+        def not_ok(self, foo: List[str] = ...) -> None:
+            print(foo)
+
+.. note::
+
+    Ellipsis expressions are legal syntax in Python 3 only. This means
+    it is not possible to elide default arguments in Python 2 code.
+    You can still elide function bodies in Python 2 by using either
+    the ``pass`` statement or by throwing a ``NotImplementedError``.
+
diff --git a/mypy/checker.py b/mypy/checker.py
index 6390b381d918..2af33533a25f 100644
--- a/mypy/checker.py
+++ b/mypy/checker.py
@@ -917,16 +917,20 @@ def check_func_def(self, defn: FuncItem, typ: CallableType, name: Optional[str])
                     item.arguments[i].variable.type = arg_type
 
                 # Type check initialization expressions.
+                body_is_trivial = self.is_trivial_body(defn.body)
                 for arg in item.arguments:
-                    if arg.initializer is not None:
-                        name = arg.variable.name()
-                        msg = 'Incompatible default for '
-                        if name.startswith('__tuple_arg_'):
-                            msg += "tuple argument {}".format(name[12:])
-                        else:
-                            msg += 'argument "{}"'.format(name)
-                        self.check_simple_assignment(arg.variable.type, arg.initializer,
-                            context=arg, msg=msg, lvalue_name='argument', rvalue_name='default')
+                    if arg.initializer is None:
+                        continue
+                    if body_is_trivial and isinstance(arg.initializer, EllipsisExpr):
+                        continue
+                    name = arg.variable.name()
+                    msg = 'Incompatible default for '
+                    if name.startswith('__tuple_arg_'):
+                        msg += "tuple argument {}".format(name[12:])
+                    else:
+                        msg += 'argument "{}"'.format(name)
+                    self.check_simple_assignment(arg.variable.type, arg.initializer,
+                        context=arg, msg=msg, lvalue_name='argument', rvalue_name='default')
 
             # Type check body in a new scope.
             with self.binder.top_frame_context():
@@ -944,11 +948,10 @@ def check_func_def(self, defn: FuncItem, typ: CallableType, name: Optional[str])
                 else:
                     return_type = self.return_types[-1]
 
-                if (not isinstance(return_type, (NoneType, AnyType))
-                        and not self.is_trivial_body(defn.body)):
+                if not isinstance(return_type, (NoneType, AnyType)) and not body_is_trivial:
                     # Control flow fell off the end of a function that was
                     # declared to return a non-None type and is not
-                    # entirely pass/Ellipsis.
+                    # entirely pass/Ellipsis/raise NotImplementedError.
                     if isinstance(return_type, UninhabitedType):
                         # This is a NoReturn function
                         self.msg.fail(message_registry.INVALID_IMPLICIT_RETURN, defn)
@@ -1000,6 +1003,21 @@ def is_unannotated_any(t: Type) -> bool:
                     self.fail(message_registry.ARGUMENT_TYPE_EXPECTED, fdef)
 
     def is_trivial_body(self, block: Block) -> bool:
+        """Returns 'true' if the given body is "trivial" -- if it contains just a "pass",
+        "..." (ellipsis), or "raise NotImplementedError()". A trivial body may also
+        start with a statement containing just a string (e.g. a docstring).
+
+        Note: functions that raise other kinds of exceptions do not count as
+        "trivial". We use this function to help us determine when it's ok to
+        relax certain checks on body, but functions that raise arbitrary exceptions
+        are more likely to do non-trivial work. For example:
+
+           def halt(self, reason: str = ...) -> NoReturn:
+               raise MyCustomError("Fatal error: " + reason, self.line, self.context)
+
+        A function that raises just NotImplementedError is much less likely to be
+        this complex.
+        """
         body = block.body
 
         # Skip a docstring
@@ -1012,7 +1030,19 @@ def is_trivial_body(self, block: Block) -> bool:
             return True
         elif len(body) > 1:
             return False
+
         stmt = body[0]
+
+        if isinstance(stmt, RaiseStmt):
+            expr = stmt.expr
+            if expr is None:
+                return False
+            if isinstance(expr, CallExpr):
+                expr = expr.callee
+
+            return (isinstance(expr, NameExpr)
+                    and expr.fullname == 'builtins.NotImplementedError')
+
         return (isinstance(stmt, PassStmt) or
                 (isinstance(stmt, ExpressionStmt) and
                  isinstance(stmt.expr, EllipsisExpr)))
diff --git a/test-data/unit/check-modules.test b/test-data/unit/check-modules.test
index cafc986d03a7..d73475603709 100644
--- a/test-data/unit/check-modules.test
+++ b/test-data/unit/check-modules.test
@@ -480,15 +480,74 @@ def f(x: int = ...) -> None: pass
 
 [case testEllipsisDefaultArgValueInStub2]
 import m
-def f(x: int = ...) -> None: pass
+def f1(x: int = ...) -> int: return 1
+def f2(x: int = '') -> int: return 1
 [file m.pyi]
-def g(x: int = '') -> None: pass
+def g1(x: int = ...) -> int: pass
+def g2(x: int = '') -> int: pass
 [out]
-tmp/m.pyi:1: error: Incompatible default for argument "x" (default has type "str", argument has type "int")
+tmp/m.pyi:2: error: Incompatible default for argument "x" (default has type "str", argument has type "int")
 main:2: error: Incompatible default for argument "x" (default has type "ellipsis", argument has type "int")
+main:3: error: Incompatible default for argument "x" (default has type "str", argument has type "int")
 
 [case testEllipsisDefaultArgValueInNonStub]
-def f(x: int = ...) -> None: pass # E: Incompatible default for argument "x" (default has type "ellipsis", argument has type "int")
+def ok_1(x: int = ...) -> None: pass
+def ok_2(x: int = ...) -> None: ...
+def ok_3(x: int = ...) -> None: raise NotImplementedError
+def ok_4(x: int = ...) -> None: raise NotImplementedError()
+def ok_5(x: int = ...) -> None:
+    """Docstring here"""
+    pass
+
+def bad_1(x: int = ...) -> None: 1   # E: Incompatible default for argument "x" (default has type "ellipsis", argument has type "int")
+def bad_2(x: int = ...) -> None:     # E: Incompatible default for argument "x" (default has type "ellipsis", argument has type "int")
+    """Docstring here"""
+    ok_1()
+def bad_3(x: int = ...) -> None:     # E: Incompatible default for argument "x" (default has type "ellipsis", argument has type "int")
+    raise Exception("Some other exception")
+[builtins fixtures/exception.pyi]
+[out]
+
+[case testEllipsisDefaultArgValueInNonStubsOverload]
+from typing import overload, Union
+
+Both = Union[int, str]
+
+@overload
+def foo(x: int, y: int = ...) -> int: ...
+@overload
+def foo(x: str, y: str = ...) -> str: ...
+def foo(x: Both, y: Both = ...) -> Both:  # E: Incompatible default for argument "y" (default has type "ellipsis", argument has type "Union[int, str]")
+    return x
+
+@overload
+def bar(x: int, y: int = ...) -> int: ...
+@overload
+def bar(x: str, y: str = ...) -> str: ...
+def bar(x: Both, y: Both = ...) -> Both:
+    raise NotImplementedError
+[builtins fixtures/exception.pyi]
+[out]
+
+[case testEllipsisDefaultArgValueInNonStubsMethods]
+from typing import Generic, TypeVar
+from typing_extensions import Protocol
+from abc import abstractmethod
+
+T = TypeVar('T')
+class Wrap(Generic[T]): ...
+
+class MyProtocol(Protocol):
+    def no_impl(self, x: Wrap[int] = ...) -> int: ...
+    def default_impl(self, x: Wrap[int] = ...) -> int: return 3  # E: Incompatible default for argument "x" (default has type "ellipsis", argument has type "Wrap[int]")
+
+class MyAbstractClass:
+    @abstractmethod
+    def no_impl(self, x: Wrap[int] = ...) -> int: raise NotImplementedError
+
+    @abstractmethod
+    def default_impl(self, x: Wrap[int] = ...) -> int: return 3  # E: Incompatible default for argument "x" (default has type "ellipsis", argument has type "Wrap[int]")
+[builtins fixtures/exception.pyi]
 [out]
 
 [case testStarImportOverlapping]
diff --git a/test-data/unit/fixtures/exception.pyi b/test-data/unit/fixtures/exception.pyi
index f9bda1b2a042..bf6d21c8716e 100644
--- a/test-data/unit/fixtures/exception.pyi
+++ b/test-data/unit/fixtures/exception.pyi
@@ -13,5 +13,11 @@ class unicode: pass
 class bool: pass
 class ellipsis: pass
 
-class BaseException: pass
+# Note: this is a slight simplification. In Python 2, the inheritance hierarchy
+# is actually Exception -> StandardError -> RuntimeError -> ...
+class BaseException:
+    def __init__(self, *args: object) -> None: ...
 class Exception(BaseException): pass
+class RuntimeError(Exception): pass
+class NotImplementedError(RuntimeError): pass
+


Code review comments:
diff_hunk: @@ -62,20 +62,81 @@ Stub file syntax
 
 Stub files are written in normal Python 3 syntax, but generally
 leaving out runtime logic like variable initializers, function bodies,
-and default arguments, or replacing them with ellipses.
+and default arguments.
 
-In this example, each ellipsis ``...`` is literally written in the
+If it is not possible to completely leave out some piece of runtime
+logic, the recommended convention is to replace or elide them with ellipsis
+expressions (``...``). Each ellipsis below is literally written in the
 stub file as three dots:
 
 .. code-block:: python
 
+    # Variables with annotations do not need to be assigned a value.
+    # So by convention, we omit them in the stub file.
     x: int
 
-    def afunc(code: str) -> int: ...
-    def afunc(a: int, b: int = ...) -> int: ...
+    # Function bodies cannot be completely removed. By convention,
+    # we replace them with `...` instead of the `pass` statement.
+    def func_1(code: str) -> int: ...
+
+    # We can do the same with default arguments.
+    def func_2(a: int, b: int = ...) -> int: ...
 
 .. note::
 
     The ellipsis ``...`` is also used with a different meaning in
     :ref:`callable types <callable-types>` and :ref:`tuple types
     <tuple-types>`.
+
+.. note::
+
+    It is always legal to use Python 3 syntax in stub files, even when
+    writing Python 2 code. The example above is a valid stub file
+    for both Python 2 and 3.
+
+Using stub file syntax at runtime
+*********************************
+
+You may also occasionally need to elide actual logic in regular
+Python code -- for example, when writing methods in
+:ref:`overload variants <function-overloading>` or
+:ref:`custom protocols <protocol-types>`.
+
+The recommended style is to use ellipses to do so, just like in
+stub files. It is also considered stylistically acceptable to
+throw a ``NotImplementedError`` in cases where the user of the
+code may accidentally call functions with no actual logic.
+
+You can also elide default arguments as long as the function body
+also contains no runtime logic: the function body only contains
+a single ellipsis, the pass statement, or a ``raise NotImplementedError()``.
+It is also ok for the function body to contain a docstring.
path: docs/source/stubs.rst
text: @user1:
```suggestion
It is also acceptable for the function body to contain a docstring.
```
"ok" is a bit too informal