Repository: python/mypy

Issue:
--disallow-any-generics should be included in --strict
If I create an alias for a generic type using a `TypeVar`, Mypy seems to infer the `TypeVar` as `Any` and lose the typing information. For example, this works:
```python
from typing import TypeVar, Callable

T = TypeVar('T')


def wrapper(func: Callable[[], T]) -> T:
    reveal_type(func)
    reveal_type(func())
    return func()
```
and gives these types:
```
mypy --strict generic_callable.py 
generic_callable.py:7: error: Revealed type is 'def () -> T`-1'
generic_callable.py:8: error: Revealed type is 'T`-1'
```
I would expect this
```python
from typing import TypeVar, Callable

T = TypeVar('T')
C = Callable[[], T]


def wrapper(func: C) -> T:
    reveal_type(func)
    reveal_type(func())
    return func()
```
to behave in the same way, but it doesn't:
```
mypy --strict generic_callable.py 
generic_callable.py:8: error: Revealed type is 'def () -> Any'
generic_callable.py:9: error: Revealed type is 'Any'
generic_callable.py:10: warning: Returning Any from function declared to return "T"
```
This can be a problem if I am using the same complex generic type multiple times, as in this real-world example:
```python
import functools
import warnings
from typing import Callable, TypeVar

_ReturnType = TypeVar('_ReturnType')
_CallableType = Callable[..., _ReturnType]


def deprecated(msg: str) -> Callable[[_CallableType], _CallableType]:
    def outer_wrapper(func: _CallableType) -> _CallableType:
        @functools.wraps(func)
        def inner_wrapper(*args: object, **kwargs: object) -> _ReturnType:
            warnings.simplefilter('always', DeprecationWarning)
            warnings.warn(f"{func.__name__} is deprecated: {msg}.",
                          category=DeprecationWarning,
                          stacklevel=2)
            warnings.simplefilter('default', DeprecationWarning)  # reset filter
            return func(*args, **kwargs)

        return inner_wrapper

    return outer_wrapper
```
which gives:
```
mypy --strict test_deprecated.py 
test_deprecated.py:18: warning: Returning Any from function declared to return "_ReturnType"
```
Having to explicitly type out `Callable[..., _ReturnType]` everywhere I use `_CallableType` would be a little cumbersome in this case.
MYPYPATH doesn't respect PEP420 namespace packages
Take this directory tree:

```
/path/to/src
└───namespace
    └───package
        │   __init__.py
        │
        └───__pycache__
                __init__.cpython-35.pyc
```

And this MYPYPATH:

```
> $env:MYPYPATH
/path/to/src
```

then running `mypy -p namespace.package` gives:

```
> mypy -p namespace.package
Can't find package 'namespace.package'
```

However, mypy works fine when you add an empty `__init__.py` into `namepace` (turning it into a regular package).



Discussion:
When you write `C = Callable[[], T]` you create a generic alias. You need to write `C[T]`, bare `C` is equivalent to `C[Any]` as with any other generic, please read the docs https://mypy.readthedocs.io/en/latest/kinds_of_types.html#type-aliases

What actually surprises me is that `--disallow-any-generics` is not in `--strict`, I think we should add it there. Do you want to make a PR?
Also unrelated suggestion, decorators like this are better typed using:
```python
F = TypeVar('F', bound=Callable[..., Any])

def deco(msg: str) -> Callable[[F], F]: ...
```
which will give you more precise types, read https://mypy.readthedocs.io/en/latest/generics.html#declaring-decorators
I know... I'm not sure what to do about this. The [algorithm from PEP 420](https://www.python.org/dev/peps/pep-0420/#specification) is quite complex (namespace packages are only considered if no "classic" package or module is found). The workaround of adding a dummy `__init__.py` is simple enough (you could also have a dummy `__init__.pyi` so that Python is not affected by this file).

I also expect that usually you'd want to create a separate set of stubs rather than type-check the original source code (though obviously this may vary per package).

I'd like to bump this. In future, there will be python programmers who never heard of pre PEP420 era of `__init__.py` and that's good. What's not good is mypy forcing them to create dummy files. 
It might get the ball rolling if somebody explained why this is such a big problem for mypy.
I just bumped into Jukka Lehtosalo and he said there shouldn't be any major issue blocking this. But given the easy workaround that exists for this, it'll likely require an outsider to write a PR.
> What's not good is mypy forcing them to create dummy files.

This issue is more complex than you think.
I don't think it's trivial, I've read your previous comment - so I don't even think I'd be able to write a PR. I just think it's an issue worth looking into eventually
I'm working on a project with a directory structure like the following:

<pre>
./a
./a/c
./a/c/file1.py
./b
./b/c
./b/c/file2.py
</pre>

Note that both, a and b, contain a subfolder called c. Adding an `__init__`  everywhere defines some modules multiple times (the actual project contains dozens folders) and things get hairy. Any progress on this issue would be much appreciated.
> Adding an `__init__` everywhere defines some modules multiple times [...]

I presume you mean in each `c` folder but not in `a` or `b` right? So `c` is a namespace package and you have both `a` and `b` in `PYTHONPATH` or `sys.path`. (And when running mypy, you pass it both `a` and `b`, which will have the same effect -- or perhaps you pass it their parent directory or the current directory.)

With that setup, I agree you have a predicament. We will have to support namespace packages in mypy. At the very least it should be possible to do something like setting `MYPYPATH=a:b` and then run `mypy -m c` and have it find both parts of `c`.
Just a philosophical note here that might influence the way mypy handles this: in my experience in a Py3-only world, packages without `__init__.py` are not a special edge case for namespace packages, they are the default case. It's pretty unreasonable to expect a new Py3 developer to remember to drop empty `__init__.py` turds everywhere when their code demonstrably runs perfectly fine without them. It's hard to even make a sensible case in code review for why they should add `__init__.py` (other than "it's the way we've always done things in the old world, get off my lawn" or "the typechecker is buggy and won't find your code without it.") The presence of `__init__.py` becomes the special case, used only when you need some package-level code or imports.

This reversal may not have been an intended consequence of PEP 420, but AFAICS it's an inevitable one.
Or perhaps this reflects your particular culture.

On Nov 25, 2017 21:48, "Carl Meyer" <notifications@github.com> wrote:

> Just a philosophical note here that might influence the way mypy handles
> this: in my experience in a Py3-only world, packages without __init__.py
> are not a special edge case for namespace packages, they are the default
> case. It's pretty unreasonable to expect a new Py3 developer to remember to
> drop empty __init__.py turds everywhere when their code demonstrably runs
> perfectly fine without them. It's hard to even make a sensible case in code
> review for why they should add __init__.py (other than "it's the way
> we've always done things in the old world, get off my lawn" or "the
> typechecker is buggy and won't find your code without it.") The presence of
> __init__.py becomes the special case, used only when you need some
> package-level code or imports.
>
> This reversal may not have been an intended consequence of PEP 420, but
> AFAICS it's an inevitable one.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/python/mypy/issues/1645#issuecomment-346986174>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ACwrMunwfsZJ03cqKrWM6ONIipLvzuh_ks5s6Pu5gaJpZM4IuKil>
> .
>

> Or perhaps this reflects your particular culture.

Certainly possible. I'd be curious to hear from another Py3-only culture
which (convincing to new Python 3 developers) arguments are used in
favor of continuing the tradition of empty `__init__.py` files. I'm not
aware of a strong practical argument that can be made for them, but if
I'm missing a good one I may start using it ;)
This seems to be hitting quite a few people, so raising priority to high.
This seems to be one the most important Python features that mypy doesn't properly support. Some potential users have told me that they were unable to use mypy because of this.
Unfortunately the work-around of adding dummy` __init__.py` files does not work when you are using tools that expect namespace packages to not have` __init__.py` files. This limitation is preventing me from getting full value from mypy.
@jfgossage As a dirty trick you can try adding dummy `__init__.pyi` files (note `i` at the end), this might work.
I am working with a structure like this:

```
project
   src
      wiseoldbird  (Python package)
          utilities (Python package)
              module1.py
              module2.py
  tests
        wiseoldbird (Python package)
             utilities (Python package)
                   test_module1.py
                   test_module2.py

```
Unfortunately if I try to use `__init__.pyi `files in all the package directories I get the following mypy error:

`tests/wiseoldbird/__init__.pyi`: error: Duplicate module named 'wiseoldbird'

As shown with #5575 I hit the same problem.  I think mypy needs to somehow handle namespace packages and be prepared for an all Python 3 world.

Resolving patch:
diff --git a/mypy/main.py b/mypy/main.py
--- a/mypy/main.py
+++ b/mypy/main.py
@@ -474,10 +474,9 @@ def add_invertible_flag(flag: str,
     disallow_any_group.add_argument(
         '--disallow-any-explicit', default=False, action='store_true',
         help='Disallow explicit Any in type positions')
-    disallow_any_group.add_argument(
-        '--disallow-any-generics', default=False, action='store_true',
-        help='Disallow usage of generic types that do not specify explicit '
-             'type parameters')
+    add_invertible_flag('--disallow-any-generics', default=False, strict_flag=True,
+                        help='Disallow usage of generic types that do not specify explicit type '
+                        'parameters', group=disallow_any_group)
 
     untyped_group = parser.add_argument_group(
         title='Untyped definitions and calls',


Test patch:
diff --git a/test-data/unit/check-flags.test b/test-data/unit/check-flags.test
--- a/test-data/unit/check-flags.test
+++ b/test-data/unit/check-flags.test
@@ -1124,3 +1124,48 @@ GroupDataDict = TypedDict(  # type: ignore
 GroupsDict = Dict[str, GroupDataDict]  # type: ignore
 [builtins fixtures/dict.pyi]
 [out]
+
+[case testCheckDefaultAllowAnyGeneric]
+from typing import TypeVar, Callable
+
+T = TypeVar('T')
+C = Callable[[], T]
+
+def f(c: C):
+    pass
+[out]
+
+[case testCheckAllowAnyGenericAnyGeneric]
+# flags: --strict --allow-any-generics
+from typing import TypeVar, Callable
+
+T = TypeVar('T')
+C = Callable[[], T]
+
+def f(c: C) -> None:
+    pass
+[out]
+
+[case testCheckDisallowAnyGenericsAnyGeneric]
+# flags: --disallow-any-generics
+from typing import TypeVar, Callable
+
+T = TypeVar('T')
+C = Callable[[], T]
+
+def f(c: C):  # E: Missing type parameters for generic type
+    pass
+[out]
+
+[case testStrictAnyGeneric]
+# flags: --strict
+from typing import TypeVar, Generic
+
+T = TypeVar('T')
+
+class A(Generic[T]):
+    pass
+
+def f(c: A) -> None:  # E: Missing type parameters for generic type
+    pass
+[out]