Repository: python/mypy

Issue:
MyPy not acknowledging __all__ 
When MyPy encounters star (*) imports it considers every symbol defined in the given module as being imported even if an `__all__` attribute is present on the module.

For example, say you have mypkg with two files, main.py:

``` python
from datetime import datetime
from mypkg.widgets import *
```

and widget.py:

``` python
from datetime import datetime

__all__ = [
    "Gadget"
]

class Gadget(object):
    pass
```

Running "mypy mypkg/app.py" issues the following error: "mypkg/app.py:2: error: Name 'datetime' already defined", though datetime should not be re-imported to the app module.

MyPy not acknowledging __all__ 
When MyPy encounters star (*) imports it considers every symbol defined in the given module as being imported even if an `__all__` attribute is present on the module.

For example, say you have mypkg with two files, main.py:

``` python
from datetime import datetime
from mypkg.widgets import *
```

and widget.py:

``` python
from datetime import datetime

__all__ = [
    "Gadget"
]

class Gadget(object):
    pass
```

Running "mypy mypkg/app.py" issues the following error: "mypkg/app.py:2: error: Name 'datetime' already defined", though datetime should not be re-imported to the app module.



Discussion:
Thanks for the report! For whoever wants to tackle this, apparently mypy ignore the value of `__all__` when computing the effect of `from ... import *`. That feels like it shouldn't be too hard to fix.

Was trying to tackle this. Given a `SymbolTableNode`, is it possible to get the variable's initializer somehow? Can't seem to figure out anything...

The initializer is not accessible via `SymbolTableNode`. One way would be to hook into the semantic analysis of assignment statements (special case `__all__` at module top level).

You probably have to special-case assignment to `__all__`. ISTR that's how
type aliases are done.

How should messes like this be handled:

``` python
from typing import List, Union

__all__ = None # type: Union[int, str, bytes, List[int], List[str]]

if True:
    __all__ = 1
elif True:
    __all__ = 'a'
elif True:
    __all__ = [1]
elif True:
    __all__ = ['a']
```

My guess is to make it an error if `__all__` has any type other than `Sequence[str]`, and, if `__all__` is assigned to multiple times, then just try to merge them all or something. So this would be an error:

``` python
__all__ = 1 # ERROR: type of __all__ must be Sequence[str]
```

and this would attempt to make `__all__` `['x', 'y']`:

``` python
if True:
    __all__ = ['x']
else:
    __all__ = ['y']
```

Any attempts to mutate it via `append` and such would be ignored (at least for now). How does all that sound?

Check the stdlib for patterns. IIRC asyncio has dynamic **all**.

But yes, it should be Sequence of stress (or Any).

--Guido (mobile)

Thanks for the report! For whoever wants to tackle this, apparently mypy ignore the value of `__all__` when computing the effect of `from ... import *`. That feels like it shouldn't be too hard to fix.

Was trying to tackle this. Given a `SymbolTableNode`, is it possible to get the variable's initializer somehow? Can't seem to figure out anything...

The initializer is not accessible via `SymbolTableNode`. One way would be to hook into the semantic analysis of assignment statements (special case `__all__` at module top level).

You probably have to special-case assignment to `__all__`. ISTR that's how
type aliases are done.

How should messes like this be handled:

``` python
from typing import List, Union

__all__ = None # type: Union[int, str, bytes, List[int], List[str]]

if True:
    __all__ = 1
elif True:
    __all__ = 'a'
elif True:
    __all__ = [1]
elif True:
    __all__ = ['a']
```

My guess is to make it an error if `__all__` has any type other than `Sequence[str]`, and, if `__all__` is assigned to multiple times, then just try to merge them all or something. So this would be an error:

``` python
__all__ = 1 # ERROR: type of __all__ must be Sequence[str]
```

and this would attempt to make `__all__` `['x', 'y']`:

``` python
if True:
    __all__ = ['x']
else:
    __all__ = ['y']
```

Any attempts to mutate it via `append` and such would be ignored (at least for now). How does all that sound?

Check the stdlib for patterns. IIRC asyncio has dynamic **all**.

But yes, it should be Sequence of stress (or Any).

--Guido (mobile)


Resolving patch:
diff --git a/mypy/checker.py b/mypy/checker.py
--- a/mypy/checker.py
+++ b/mypy/checker.py
@@ -431,6 +431,15 @@ def visit_file(self, file_node: MypyFile, path: str) -> None:
         self.errors.set_ignored_lines(set())
         self.current_node_deferred = False
 
+        all_ = self.globals.get('__all__')
+        if all_ is not None and all_.type is not None:
+            seq_str = self.named_generic_type('typing.Sequence',
+                                              [self.named_type('builtins.str')])
+            if not is_subtype(all_.type, seq_str):
+                str_seq_s, all_s = self.msg.format_distinctly(seq_str, all_.type)
+                self.fail(messages.ALL_MUST_BE_SEQ_STR.format(str_seq_s, all_s),
+                          all_.node)
+
     def check_second_pass(self):
         """Run second pass of type checking which goes through deferred nodes."""
         self.pass_num = 1
diff --git a/mypy/messages.py b/mypy/messages.py
--- a/mypy/messages.py
+++ b/mypy/messages.py
@@ -81,6 +81,7 @@
 ARGUMENT_TYPE_EXPECTED = "Function is missing a type annotation for one or more arguments"
 KEYWORD_ARGUMENT_REQUIRES_STR_KEY_TYPE = \
     'Keyword argument only valid with "str" key type in call to "dict"'
+ALL_MUST_BE_SEQ_STR = 'Type of __all__ must be {}, not {}'
 
 
 class MessageBuilder:
diff --git a/mypy/semanal.py b/mypy/semanal.py
--- a/mypy/semanal.py
+++ b/mypy/semanal.py
@@ -222,6 +222,7 @@ def __init__(self,
         self.check_untyped_defs = check_untyped_defs
         self.postpone_nested_functions_stack = [FUNCTION_BOTH_PHASES]
         self.postponed_functions_stack = []
+        self.all_exports = set()  # type: Set[str]
 
     def visit_file(self, file_node: MypyFile, fnam: str) -> None:
         self.errors.set_file(fnam)
@@ -250,6 +251,11 @@ def visit_file(self, file_node: MypyFile, fnam: str) -> None:
 
         self.errors.set_ignored_lines(set())
 
+        if '__all__' in self.globals:
+            for name, g in self.globals.items():
+                if name not in self.all_exports:
+                    g.module_public = False
+
     def visit_func_def(self, defn: FuncDef) -> None:
         phase_info = self.postpone_nested_functions_stack[-1]
         if phase_info != FUNCTION_SECOND_PHASE:
@@ -1069,6 +1075,11 @@ def visit_assignment_stmt(self, s: AssignmentStmt) -> None:
         self.process_typevar_declaration(s)
         self.process_namedtuple_definition(s)
 
+        if (len(s.lvalues) == 1 and isinstance(s.lvalues[0], NameExpr) and
+                s.lvalues[0].name == '__all__' and s.lvalues[0].kind == GDEF and
+                isinstance(s.rvalue, (ListExpr, TupleExpr))):
+            self.add_exports(*s.rvalue.items)
+
     def check_and_set_up_type_alias(self, s: AssignmentStmt) -> None:
         """Check if assignment creates a type alias and set it up as needed."""
         # For now, type aliases only work at the top level of a module.
@@ -1622,6 +1633,9 @@ def visit_operator_assignment_stmt(self,
                                        s: OperatorAssignmentStmt) -> None:
         s.lvalue.accept(self)
         s.rvalue.accept(self)
+        if (isinstance(s.lvalue, NameExpr) and s.lvalue.name == '__all__' and
+                s.lvalue.kind == GDEF and isinstance(s.rvalue, (ListExpr, TupleExpr))):
+            self.add_exports(*s.rvalue.items)
 
     def visit_while_stmt(self, s: WhileStmt) -> None:
         s.expr.accept(self)
@@ -1837,6 +1851,17 @@ def visit_call_expr(self, expr: CallExpr) -> None:
             for a in expr.args:
                 a.accept(self)
 
+            if (isinstance(expr.callee, MemberExpr) and
+                    isinstance(expr.callee.expr, NameExpr) and
+                    expr.callee.expr.name == '__all__' and
+                    expr.callee.expr.kind == GDEF and
+                    expr.callee.name in ('append', 'extend')):
+                if expr.callee.name == 'append' and expr.args:
+                    self.add_exports(expr.args[0])
+                elif (expr.callee.name == 'extend' and expr.args and
+                        isinstance(expr.args[0], (ListExpr, TupleExpr))):
+                    self.add_exports(*expr.args[0].items)
+
     def translate_dict_call(self, call: CallExpr) -> Optional[DictExpr]:
         """Translate 'dict(x=y, ...)' to {'x': y, ...}.
 
@@ -2206,6 +2231,11 @@ def add_local(self, node: Union[Var, FuncBase], ctx: Context) -> None:
         node._fullname = name
         self.locals[-1][name] = SymbolTableNode(LDEF, node)
 
+    def add_exports(self, *exps: Node) -> None:
+        for exp in exps:
+            if isinstance(exp, StrExpr):
+                self.all_exports.add(exp.value)
+
     def check_no_global(self, n: str, ctx: Context,
                         is_func: bool = False) -> None:
         if n in self.globals:


Test patch:
diff --git a/mypy/test/data/check-modules.test b/mypy/test/data/check-modules.test
--- a/mypy/test/data/check-modules.test
+++ b/mypy/test/data/check-modules.test
@@ -392,6 +392,32 @@ class B: pass
 x = A()
 y = B()
 
+[case testExportedValuesInImportAll]
+import typing
+from m import *
+_ = a
+_ = b
+_ = c
+_ = d
+_ = e
+_ = f # E: Name 'f' is not defined
+_ = _g # E: Name '_g' is not defined
+[file m.py]
+__all__ = ['a']
+__all__ += ('b',)
+__all__.append('c')
+__all__.extend(('d', 'e'))
+
+a = b = c = d = e = f = _g = 1
+[builtins fixtures/module_all.py]
+
+[case testAllMustBeSequenceStr]
+import typing
+__all__ = [1, 2, 3]
+[builtins fixtures/module_all.py]
+[out]
+main: error: Type of __all__ must be Sequence[str], not List[int]
+
 [case testEllipsisInitializerInStubFileWithType]
 import m
 m.x = '' # E: Incompatible types in assignment (expression has type "str", variable has type "int")
diff --git a/mypy/test/data/fixtures/module_all.py b/mypy/test/data/fixtures/module_all.py
new file mode 100644
--- /dev/null
+++ b/mypy/test/data/fixtures/module_all.py
@@ -0,0 +1,15 @@
+from typing import Generic, Sequence, TypeVar
+_T = TypeVar('_T')
+
+class object:
+    def __init__(self) -> None: pass
+class module: pass
+class type: pass
+class function: pass
+class int: pass
+class str: pass
+class list(Generic[_T], Sequence[_T]):
+    def append(self, x: _T): pass
+    def extend(self, x: Sequence[_T]): pass
+    def __add__(self, rhs: Sequence[_T]) -> list[_T]: pass
+class tuple: pass