Repository: python/mypy

Issue:
Support Callable[..., t] (with literal ellipsis, unspecified argument types and specified return type)
It would be handy to be able to specify a callable whose argument types are not checked but whose return type is, e.g., by specifying the argument types as `None` instead of a list. 

For example, consider `map`:

```
def map(function, *iterables):
```

How would you annotate that? Trying to actually specify the argument types to `function` to match the element types of the iterables in `iterables` would require something horrible like C++11's parameter packs and variable-length type tuples. But just specifying `Function` means there's no way to know that the result iterates values of the same type the function returns, which seems like a pity.

If you could just use `None` or `Any` instead of a list, you could do this:

```
def map(function: Function[None, T], *iterables: Iterable[Iterable]) -> Iterator[T]:
```



Discussion:
I notice that in the existing [stubs](https://github.com/JukkaL/mypy/blob/master/stubs/3.2/builtins.py#L744) you've tried to annotate `map` like this:

```
# TODO more than two iterables
@overload
def map(func: Function[[T1], S], iter1: Iterable[T1]) -> Iterator[S]: pass
@overload
def map(func: Function[[T1, T2], S],
        iter1: Iterable[T1],
        iter2: Iterable[T2]) -> Iterator[S]: pass
```

This is similar to the way people used to do things in C++03. Boost even came with a `preprocessor` library that helped you generate 10 overloads for your template, with 0 through 9 parameters. But this was a huge mess. And, as I noted above, I don't really think we want the C++11 solution (which would require using a template parameter pack to capture the types of `iterable`, then forwarding through a recursive function template to extract the `T` out of each `Iterable[T]`, and another recursive function template to pack them back up into a single type tuple).

I think the easy answer is just to not try to fully annotate all possibilities. Yes, it means that `map((lambda x, y, z: x+y+z), file1, file2, range(100))` would pass the type check even though it shouldn't, but how often is that going to come up?

You can just leave out types of (some) arguments or give them `Any` types (which means the same thing). Variables with `Any` types are dynamically typed and are compatible with everything. For example:

```
from typing import Any
def f(x: Any) -> int: return 1
f(1)  # Ok
f('x')  # Ok
f(1) + 'x'  # Not ok, can't add int and str
```

My gut feeling is that it makes sense to have ugly annotations for `map` and `zip` since they are in builtins and they are used pretty often in the wild. Also, by not properly annotating the arguments the return values would have imprecise types, which would propagate elsewhere via type inference, and this would be a shame.

We should probably have a fallback overload variant that takes *args to cover all cases (even if this would imply less precise types).

However, for less frequently used modules/functions falling back to dynamic typing is probably the most reasonable approach. I.e., we'd consider `map`/`zip` as exceptions/legacy that we just need to be able to type check properly, even if it's hacky or does not work in every case.

Another argument is that one the first examples that a programmer with a functional programming background tries to type check uses `map`, and we'd give a pretty poor first impression if we couldn't type check that... :-)

> You can just leave out types of (some) arguments or give them Any types (which means the same thing).

You're missing the point here. (Probably my fault.) I'm asking how to declare the types of arguments (or return values) that are themselves functions.

This is why I used `map` as an example, because it's the most familiar higher-order function. Its first argument is a function. The argument list of that function is dependent on the other arguments to `map`. That makes it very hard to specify.

> Also, by not properly annotating the arguments the return values would have imprecise types, which would propagate elsewhere via type inference, and this would be a shame.

That's exactly my point. Sometimes, it's hard, verbose, or even impossible to annotate the argument types of the function argument, but easy to annotate the return type of the function argument. And often that's sufficient to provide a precise return type for the higher-order function.

Again, look at `map`. If all I know is that its first argument is a `Function[???, int]`, that's enough to tell me that its return type is `int`.

So, it's useful to be able to specify an argument type that means `Function[???, int]` in some way, which I suggested spelling `Function[None, int]`.

Ah, sorry, I misunderstood your point. There is currently no way to do this. I guess we could have extra syntax for something like this, such as `Function[[...], int]`, though maybe that would be abusing '`...`' (and it wouldn't work in Python 2).

More generally, callable types are restricted to a fixed number of arguments and there's no way to have keyword arguments, `*args*`, `**kwargs`, or arguments with default values in callable types. This would be technically simple but it's tricky to come up with a reasonable syntax.

Here are some (maybe stupid) ideas:

```
Function[[int, [str]], None]  # default arg value
Function[[int, vararg(int)], None]  # *args
Function[args(str, x=int, *int), None]  # default value (usable as keyword arg) and *args
Function[[('x=', 'int), ('*', int)], None]  # default arg value (usable as keyword arg) and *args
```

I wasn't even trying to solve the fully general problem, because I was assuming that it's unsolvable, or at least not worth solving. If you _do_ want to solve it, it's even worse than you're thinking. Because of keyword arguments, even the parameter _names_ can matter, not just the number and types. So you need something at least as complex as `inspect.Signature` and `inspect.ArgSpec` to determine whether a function is actually callable with a set of arguments. But that also points at the possible solution: For anything more complicated than a fixed list of arguments, just take an inspect.Signature. (I'm not sure even that's sufficient, but if not, it would be easier to build from there than to start from scratch and try to cover all the cases that it's had to tackle that you're probably not thinking of yet…)

Getting this right is part of why programming in Swift, or doing compile-time metaprogramming in C++, is not as much fun as programming in Python. ML and Haskell solve this by just not allowing anything fancy. In Haskell, all functions take one argument; if you need more, you either take a tuple, or you write it curried (which are of course equivalent). That makes rigorously defining the type of any function dead simple. Too bad that won't work for Python (or C++ or Swift).

But maybe the complicated cases don't need to be solved, as long as the simple ones can be.

I think my proposal of just using `None`, or maybe `Any`, in place of the list of arguments is good enough for the case I'm worried about:

```
Function[None, int] # function that takes who-knows-what but definitely returns int
```

There's one intermediate case that seems potentially important: I want a function that takes an `int` and a `str`. It may take other positional or keyword-only parameters with default values; it may have `*args` or `**kwargs`; it may actually only take an `int` and a `*args` that will accept my `str`; I don't care, as long as I can call it with an `int` and a `str`. Is that what `Function[[int, str], None]` means, or does that mean specifically a function that must take an `int` and a `str` and nothing else?

`Function[[int, str], None]` means a function that can be called with two positional arguments, the first with type `int` and the second with type `str`. 

The current syntax seems to cover the majority of use cases for callable values. I was aware of the potential complexity and that's why the current annotation syntax is so simplistic.

Another option would be to use `Function[Any, int]` for a function that is compatible with arbitrary argument lists but that must return `int`. I actually kinda like this, though it could be potentially confusing.

`map` is tricky because the function is often a lambda, and we need to infer the return type. Additionally, the return type often depends on the argument type(s), which must be inferred as well. 

The PEP 484 draft uses `Callable[..., T]` (with literal `...`) for a callable type with arbitrary arguments but with a specific return type `T`. Mypy should support that.


Resolving patch:
diff --git a/mypyc/genops.py b/mypyc/genops.py
--- a/mypyc/genops.py
+++ b/mypyc/genops.py
@@ -16,9 +16,9 @@ def f(x: int) -> int:
 from typing import (
     TypeVar, Callable, Dict, List, Tuple, Optional, Union, Sequence, Set, Any, cast
 )
-from typing_extensions import overload, ClassVar, NoReturn
+from typing_extensions import overload, NoReturn
+from collections import OrderedDict
 from abc import abstractmethod
-import sys
 import importlib.util
 import itertools
 
@@ -192,7 +192,7 @@ def build_ir(modules: List[MypyFile],
         builder = IRBuilder(types, graph, errors, mapper, module_names, pbv, options)
         builder.visit_mypy_file(module)
         module_ir = ModuleIR(
-            builder.imports,
+            list(builder.imports),
             builder.functions,
             builder.classes,
             builder.final_names
@@ -1050,7 +1050,10 @@ def __init__(self,
 
         self.errors = errors
         self.mapper = mapper
-        self.imports = []  # type: List[str]
+        # Notionally a list of all of the modules imported by the
+        # module being compiled, but stored as an OrderedDict so we
+        # can also do quick lookups.
+        self.imports = OrderedDict()  # type: OrderedDict[str, None]
 
     def visit_mypy_file(self, mypyfile: MypyFile) -> None:
         if mypyfile.fullname() in ('typing', 'abc'):
@@ -1518,29 +1521,8 @@ def allocate_class(self, cdef: ClassDef) -> None:
                           [self.load_globals_dict(), self.load_static_unicode(cdef.name),
                            tp], cdef.line)
 
-    # An unfortunate hack: for some stdlib modules, pull in modules
-    # that the stubs reexport things from. This works around #393
-    # in these cases.
-    import_maps = {
-        'os': tuple(['os.path'] + ([] if sys.platform == 'win32' else ['posix'])),
-        'os.path': ('os',),
-        'tokenize': ('token',),
-        'weakref': ('_weakref',),
-        'asyncio': ('asyncio.events', 'asyncio.tasks',),
-        'click': ('click.core', 'click.termui', 'click.decorators',
-                  'click.exceptions', 'click.types'),
-        'ast': ('_ast',),
-    }  # type: ClassVar[Dict[str, Sequence[str]]]
-
     def gen_import(self, id: str, line: int) -> None:
-        if id in IRBuilder.import_maps:
-            for dep in IRBuilder.import_maps[id]:
-                self._gen_import(dep, line)
-
-        self._gen_import(id, line)
-
-    def _gen_import(self, id: str, line: int) -> None:
-        self.imports.append(id)
+        self.imports[id] = None
 
         needs_import, out = BasicBlock(), BasicBlock()
         first_load = self.add(LoadStatic(object_rprimitive, 'module', id))
@@ -2817,7 +2799,7 @@ def visit_name_expr(self, expr: NameExpr) -> Value:
             if value is not None:
                 return value
 
-        if isinstance(expr.node, MypyFile):
+        if isinstance(expr.node, MypyFile) and expr.node.fullname() in self.imports:
             return self.load_module(expr.node.fullname())
 
         # If the expression is locally defined, then read the result from the corresponding
@@ -2853,11 +2835,11 @@ def visit_member_expr(self, expr: MemberExpr) -> Value:
             if value is not None:
                 return value
 
-        if self.is_module_member_expr(expr):
-            return self.load_module_attr(expr)
-        else:
-            obj = self.accept(expr.expr)
-            return self.get_attr(obj, expr.name, self.node_type(expr), expr.line)
+        if isinstance(expr.node, MypyFile) and expr.node.fullname() in self.imports:
+            return self.load_module(expr.node.fullname())
+
+        obj = self.accept(expr.expr)
+        return self.get_attr(obj, expr.name, self.node_type(expr), expr.line)
 
     def get_attr(self, obj: Value, attr: str, result_type: RType, line: int) -> Value:
         if (isinstance(obj.type, RInstance) and obj.type.class_ir.is_ext_class
@@ -5210,7 +5192,8 @@ def load_global(self, expr: NameExpr) -> Value:
         """
         # If the global is from 'builtins', turn it into a module attr load instead
         if self.is_builtin_ref_expr(expr):
-            return self.load_module_attr(expr)
+            assert expr.node, "RefExpr not resolved"
+            return self.load_module_attr_by_fullname(expr.node.fullname(), expr.line)
         if (self.is_native_module_ref_expr(expr) and isinstance(expr.node, TypeInfo)
                 and not self.is_synthetic_type(expr.node)):
             assert expr.fullname is not None
@@ -5257,10 +5240,6 @@ def load_static_unicode(self, value: str) -> Value:
     def load_module(self, name: str) -> Value:
         return self.add(LoadStatic(object_rprimitive, 'module', name))
 
-    def load_module_attr(self, expr: RefExpr) -> Value:
-        assert expr.node, "RefExpr not resolved"
-        return self.load_module_attr_by_fullname(expr.node.fullname(), expr.line)
-
     def load_module_attr_by_fullname(self, fullname: str, line: int) -> Value:
         module, _, name = fullname.rpartition('.')
         left = self.load_module(module)


Test patch:
diff --git a/mypyc/test-data/run.test b/mypyc/test-data/run.test
--- a/mypyc/test-data/run.test
+++ b/mypyc/test-data/run.test
@@ -4645,3 +4645,31 @@ from native import foo
 
 assert foo(None) == None
 assert foo([1, 2, 3]) == ((1, 2, 3), [1, 2, 3])
+
+[case testReexport]
+# Test that we properly handle accessing values that have been reexported
+import a
+def f(x: int) -> int:
+    return a.g(x) + a.foo + a.b.foo
+
+whatever = a.A()
+
+[file a.py]
+from b import g as g, A as A, foo as foo
+import b
+
+[file b.py]
+def g(x: int) -> int:
+    return x + 1
+
+class A:
+    pass
+
+foo = 20
+
+[file driver.py]
+from native import f, whatever
+import b
+
+assert f(20) == 61
+assert isinstance(whatever, b.A)