Repository: python/mypy

Issue:
Handling module names ending with `__init__`
**Bug Report**

Attempts to find type hints that import package submodules whose names end with `__init__` seem to prevented. An example being `test/test__init__.py`, with double underscores on both ends.

**To Reproduce**

I tried to create a script for this, but roughly, it does the following:

1. Create a package directory.
2. Create some `*__init__.py` and `__init__.py`. Some other combination of underscores and names can also be checked.
3. Import the former from the latter, using either absolute or relative.
4. Type check the package using mypy.

A Bash script for this:

```bash
#!/usr/bin/bash

# Create a namespace package directory.
package_name='test'
if [[ -e "${package_name}" ]]; then
  printf 'Path %s exists. Needed for tests.\n' "$package_name"
  read -n1 -r \
    -p 'Press <Y> to remove and remake. Press anything else to abort.' \
    key
  if [[ 'Y' != "$key" ]]; then
    printf 'Aborting. Pressed key was: %s\n' "$key"
    exit 1
  fi
  rm -dfr test
fi
mkdir test

# Create submodules to import.
module_names=(     \
  'test_init__'    \
  'test_init___'   \
  'test__init__'   \
  'test__init___'  \
  'test___init__'  \
  'test___init___' \
)
# Make up some variables to be imported.
for name in "${module_names[@]}"; do
  printf 'num_%s = 1' "${name}" > "${package_name}/${name}.py"
done

# Relative import of variables.
cat << EOF > "${package_name}/__init__.py"
from .test_init__ import num_test_init__
from .test_init___ import num_test_init___
from .test__init__  import num_test__init__ # Errors
from .test__init___ import num_test__init___
from .test___init__  import num_test___init__ # Errors
from .test___init___ import num_test___init___
EOF
python3 -m "${package_name}.__init__"  # Loads fine in Python.
python3 -m mypy --config-file= --package "${package_name}"
# Output:
# test/__init__.py:3: error: Skipping analyzing 'test.test__init__': found module but no type hints or library stubs
# test/__init__.py:3: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
# test/__init__.py:5: error: Skipping analyzing 'test.test___init__': found module but no type hints or library stubs
# Found 2 errors in 1 file (checked 5 source files)
```

Relative imports of modules produce slightly different errors, but they are errors nonetheless.
```bash
cat << EOF > "${package_name}/__init__.py"
from . import test_init__
from . import test_init___
from . import test__init__  # Errors
from . import test__init___
from . import test___init__  # Errors
from . import test___init___
EOF
python3 -m "${package_name}.__init__"  # Loads fine in Python.
python3 -m mypy --config-file= --package "${package_name}"
# Output:
# test/__init__.py:3: error: Module 'test' has no attribute 'test__init__'; maybe "test__init___", "test___init__", or "test_init__"?
# test/__init__.py:5: error: Module 'test' has no attribute 'test___init__'; maybe "test___init___", "test__init__", or "test__init___"?
# Found 2 errors in 1 file (checked 5 source files)
```

Type checking with absolute imports exhibit similar behaviours.

**Expected Behavior**

I expected the type checks to pass. I am not aware of module naming restrictions that might restrict this, but there might be some? Naming might be too much of an edge case to support even if not?

**Actual Behavior**

Output from mypy noted at the end of the Bash scripts.

**Your Environment**

- Mypy version used: `mypy 0.812` from `python3 -m mypy --version`
- Mypy command-line flags: `python3 -m mypy --config-file= --package "${package_name}"`
- Mypy configuration options from `mypy.ini` (and other config files): Not applicable with `--config-file=`.
- Python version used: `Python 3.9.2` from `python3 --version`
- Operating system and version: `Description:    Debian GNU/Linux bullseye/sid` from `lsb_release --description`


Discussion:
There's probably an incorrect `endswith("__init__")` check somewhere. This should be easy to fix once you find it (I'd grep for `__init__` in the source code), so feel free to submit a PR!
Thank you for your advice. You are spot on in regards to the method used in the condtion check!

I found one relevant to my specific case. Using `os.path.basename(path) in ('__init__.py', '__init__.pyi')` instead in the following snippet allows my Bash script test to pass.
<https://github.com/python/mypy/blob/2ac722ac7d844f4c5ae2f3c36f3470543cf01b11/mypy/modulefinder.py#L494-L497>
A problem here is that I cannot tell _why_ this works, through what call chain, to have fixed this. Reading the docstring, I could not tell why the function was called. In particular, this means I do not know how to write a regression test for this. I made [two attempts](https://github.com/BoniLindsley/mypy/commit/f0d3b2312dbcb2706abf62085118ac7250661463) in the corresponding test file, following the test patterns in there, but they do not actually fail without the fix. So the tests were wrong.

There are other parts that use a similar check, such as
<https://github.com/python/mypy/blob/2ac722ac7d844f4c5ae2f3c36f3470543cf01b11/mypy/modulefinder.py#L429-L437>
While this likely should be changed as well, I am not even sure what the method does, so I am reluctant to change it, especially not without a failing test first.

So, all in all, even though I have found a possible fix, I am not sure how to proceed from here.

Resolving patch:
diff --git a/mypy/modulefinder.py b/mypy/modulefinder.py
--- a/mypy/modulefinder.py
+++ b/mypy/modulefinder.py
@@ -369,9 +369,7 @@ def _can_find_module_in_parent_dir(self, id: str) -> bool:
             self.options,
             stdlib_py_versions=self.stdlib_py_versions,
         )
-        while any(
-            file.endswith(("__init__.py", "__init__.pyi")) for file in os.listdir(working_dir)
-        ):
+        while any(is_init_file(file) for file in os.listdir(working_dir)):
             working_dir = os.path.dirname(working_dir)
             parent_search.search_paths = SearchPaths((working_dir,), (), (), ())
             if not isinstance(parent_search._find_module(id, False), ModuleNotFoundReason):
@@ -585,7 +583,7 @@ def find_modules_recursive(self, module: str) -> List[BuildSource]:
         sources = [BuildSource(module_path, module, None)]
 
         package_path = None
-        if module_path.endswith(("__init__.py", "__init__.pyi")):
+        if is_init_file(module_path):
             package_path = os.path.dirname(module_path)
         elif self.fscache.isdir(module_path):
             package_path = module_path
@@ -648,9 +646,13 @@ def matches_exclude(
     return False
 
 
+def is_init_file(path: str) -> bool:
+    return os.path.basename(path) in ("__init__.py", "__init__.pyi")
+
+
 def verify_module(fscache: FileSystemCache, id: str, path: str, prefix: str) -> bool:
     """Check that all packages containing id have a __init__ file."""
-    if path.endswith(("__init__.py", "__init__.pyi")):
+    if is_init_file(path):
         path = os.path.dirname(path)
     for i in range(id.count(".")):
         path = os.path.dirname(path)
@@ -664,7 +666,7 @@ def verify_module(fscache: FileSystemCache, id: str, path: str, prefix: str) ->
 
 def highest_init_level(fscache: FileSystemCache, id: str, path: str, prefix: str) -> int:
     """Compute the highest level where an __init__ file is found."""
-    if path.endswith(("__init__.py", "__init__.pyi")):
+    if is_init_file(path):
         path = os.path.dirname(path)
     level = 0
     for i in range(id.count(".")):


Test patch: