Repository: pypa/setuptools

Issue:
[FR] Include type information by default (`*.pyi`, `py.typed`) - PEP 561
### What's the problem this feature will solve?

[PEP 561](https://www.python.org/dev/peps/pep-0561/) specifies how type information should be included in a distributions. Namely package authors may choose to add `py.typed` files to fully typed packages and / or include `.pyi` stub files. By default, those files won't be included in the sdist / wheel distribution packages.

To add them, a package author needs to set `include_package_data = True` and include them via the `MANIFEST.in` or with `options.package_data` (eg. with `* = py.typed, *.pyi`). Although there are some great guides out there explaining it, I believe we could make this much simpler.

https://blog.whtsky.me/tech/2021/dont-forget-py.typed-for-your-typed-python-package/
https://jugmac00.github.io/blog/bite-my-shiny-type-annotated-library/

### Describe the solution you'd like

Add a new option `include_typing_files` or `include_typing_information` as a shorthand for `options.package_data`-> `* = py.typed, *.pyi`.

**How should it be used?**
Authors wanting to include typing files (`py.typed` / stub files) should set ~both `include_package_data = True` and~ `include_typing_files = True`.

**How would it work with other options to include package data files?**
Conceptually this option would add the `* = py.typed, *.pyi` pattern to `options.package_data`. This also means any file matched by `exclude_package_data` should still be excluded.

### Alternative Solutions

~Don't require `include_package_data = True`. Similar to license files, one could argue if an author sets `include_typing_files = True`, they already expressed what they want to do. No need to require an additional option.~

Include `py.typed` and `*.pyi` files by default.

### Additional context

If others agree that this is something worth adding, I could work on a PR for it.

### Code of Conduct

- [X] I agree to follow the PSF Code of Conduct


Discussion:
Hi @cdce8p thank you very much for bringing this issue for discussion.

My personal opinion (which might not be shared by the `setuptools` maintainers) is that it is very unlikely that a new configuration option will be added. There is already a lot of options and people seem to have a hard time understanding/dealing with them.

Something that is more likely to happen would besetuptools starting to includ `*.pyi` files by default. Would that work for you?

One way to achieve that would be overriding the `setuptools.commands.sdist.add_default` method (or specifically `_add_defaults_python`) to add the type stubs on top of what is done in `distutils.command.sdist`:

https://github.com/pypa/setuptools/blob/6f9e2fe945094ccf6d787bcc1464a4b17e700a44/setuptools/_distutils/command/sdist.py#L210-L322

Please also note that there is a related pending issue in `distutils` right now that would also improve the user experience when working with type stubs: https://github.com/pypa/distutils/issues/108
Thanks for your comment @abravalheri!

> Something that is more likely to happen would be setuptools starting to includ `*.pyi` files by default. Would that work for you?

Personally I'm more interested in `py.typed` tbh but including both types (`*.pyi` and `py.typed`) by default would certainly work. `poetry` and `flit` both do that although only because they just include all files within package directories.
> Personally I'm more interested in py.typed tbh but including both types (*.pyi and py.typed)

Oh yeah, I forgot about `py.typed`. I think including both by default would be ideal. Thanks for reminding me.

Would you be interested in working in a PR? There is no rush for that, but any help would be more than welcome!
> Would you be interested in working in a PR? There is no rush for that, but any help would be more than welcome!

At the moment, I don't have the bandwidth for it unfortunately. If someone else would like to pick this up, that would be awesome.
I am currently facing an issue distributing `*.pyi` files along my extensions.
For multiple simple projects I only have `ext_modules=` or `py_modules=` set. (no package folder needed for a one file project)

This seemed to work in the past, but I recently figured out that on some platforms the .pyi file is missing.

```py
setup(
  name='mymodule',
  ext_modules=[Extension(name='mymodule', ...)],
  data_files=[('.', ['mymodule.pyi'])],
)
```

This code was indeed from _"some great guides out there explaining it"_, but is broken as the '.' is not a valid package.
The source and wheel distributions will contain the data file but when installed the data file will be missing.

The expected installed version of the project should countain two files in the `site-packages` folder:
- `mymodule.___.pyd` or `mymodule.___.so`
- `mymodule.pyi`
 
 [Here](https://github.com/szabolcsdombi/zengl/blob/67afe6f6ee96a2cfef8d6a299869b4b36675573d/setup.py#L26-L28) I have a project with a similar setup.

---

I am interested in making the changes necessary. 

> One way to achieve that would be overriding the setuptools.commands.sdist.add_default method (or specifically _add_defaults_python) to add the type stubs on top of what is done in distutils.command.sdist:

Is this still the way to go?

[Here](https://github.com/szabolcsdombi/setuptools/pull/1/files) is a proof for no actual call is made to `setuptools/setuptools/_distutils/command/sdist.py:_add_defaults_python()`


Based on the advice above, I decided to try modifying
https://github.com/pypa/setuptools/blob/71b355ba8fcbb9341364699375ee786100b3cf75/setuptools/_distutils/command/build_py.py#L302-L303
to this
```py
def get_source_files(self):
    py_files = [module[-1] for module in self.find_all_modules()]

    possible_stub_files = set(os.path.splitext(f)[0] + ".pyi" for f in py_files)
    stub_files = [f for f in possible_stub_files if os.path.isfile(f)]

    possible_py_typed_files = set(os.path.dirname(f) + "/py.typed" for f in py_files)
    py_typed_files = [f for f in possible_py_typed_files if os.path.isfile(f)]

    return py_files + stub_files + py_typed_files
```
I wasn't sure how to test this, but I tested it in the following way and it was successful:
- I created the following directory structure (note that pyproject.toml just contained project name, version, requires_python and the build-system info, so not the usual "py.typed" stuff):
  ```
  pyproject.toml
  setup.py
  src/module_name/__init__.py
  src/module_name/py.typed
  src/module_name/file.py
  src/module_name/file.pyi
  ```
- I created a venv and installed with `pip install -e` this git repository with my changes shown above. Then I install mypy and wheel as well.
- Then I ran `python setup.py bdist_wheel` and ran `pip install` on the wheel in `dist`.
- I created `test_file.py` in the root repository and just put `import module_name` and then ran mypy on it and got no warnings.
When I test the exact same method but without my code changes, mypy reports errors. This test also works if `src/module_name` is replaced by `module_name`, and for the mypy test stage you need to put `test_file.py` in a new folder and run mypy from there so that it detects the installation rather than the source files.

I tried creating a test for this by modifying
https://github.com/pypa/setuptools/blob/71b355ba8fcbb9341364699375ee786100b3cf75/setuptools/_distutils/tests/test_sdist.py#L164
by adding lines to create some `py.typed` and `.pyi` files and adding more entries to `expected` but when I run `tox` I don't think this test runs. (Also when I run tox I always get one failing test that seems to be to do with an error with pip installing something to do with `@git+https://...`)
Hi @Danie-1, would you like to provide a PR for that? We can iterate over it and iron out the issues...

Just one thing, for an eventual PR, it would be important to modify (setuptools/command/build_py.py)(https://github.com/pypa/setuptools/blob/main/setuptools/command/build_py.py] instead of `_distutils` (you can use inheritance to overwrite methods).

The reason why we should avoid modifying files under `_distutils` is that this directory is maintained externally in https://github.com/pypa/distutils.

> Also when I run tox I always get one failing test that seems to be to do with an error with pip installing something to do with @git+https://...

Yeah, sorry this is unfortunate. You can try running `tox -- -p no:perf` to disable the performance tests that try to git clone from github.

Resolving patch:
diff --git a/setuptools/command/build_py.py b/setuptools/command/build_py.py
--- a/setuptools/command/build_py.py
+++ b/setuptools/command/build_py.py
@@ -16,6 +16,9 @@
 from ..warnings import SetuptoolsDeprecationWarning
 
 
+_IMPLICIT_DATA_FILES = ('*.pyi', 'py.typed')
+
+
 def make_writable(target):
     os.chmod(target, os.stat(target).st_mode | stat.S_IWRITE)
 
@@ -116,6 +119,7 @@ def find_data_files(self, package, src_dir):
             self.package_data,
             package,
             src_dir,
+            extra_patterns=_IMPLICIT_DATA_FILES,
         )
         globs_expanded = map(partial(glob, recursive=True), patterns)
         # flatten the expanded globs into an iterable of matches
@@ -285,7 +289,7 @@ def exclude_data_files(self, package, src_dir, files):
         return list(unique_everseen(keepers))
 
     @staticmethod
-    def _get_platform_patterns(spec, package, src_dir):
+    def _get_platform_patterns(spec, package, src_dir, extra_patterns=[]):
         """
         yield platform-specific path patterns (suitable for glob
         or fn_match) from a glob-based spec (such as
@@ -293,6 +297,7 @@ def _get_platform_patterns(spec, package, src_dir):
         matching package in src_dir.
         """
         raw_patterns = itertools.chain(
+            extra_patterns,
             spec.get('', []),
             spec.get(package, []),
         )


Test patch:
diff --git a/setuptools/tests/test_build_meta.py b/setuptools/tests/test_build_meta.py
--- a/setuptools/tests/test_build_meta.py
+++ b/setuptools/tests/test_build_meta.py
@@ -372,8 +372,10 @@ def test_build_with_pyproject_config(self, tmpdir, setup_script):
             "src": {
                 "foo": {
                     "__init__.py": "__version__ = '0.1'",
+                    "__init__.pyi": "__version__: str",
                     "cli.py": "def main(): print('hello world')",
                     "data.txt": "def main(): print('hello world')",
+                    "py.typed": "",
                 }
             },
         }
@@ -406,8 +408,10 @@ def test_build_with_pyproject_config(self, tmpdir, setup_script):
             'foo-0.1/src',
             'foo-0.1/src/foo',
             'foo-0.1/src/foo/__init__.py',
+            'foo-0.1/src/foo/__init__.pyi',
             'foo-0.1/src/foo/cli.py',
             'foo-0.1/src/foo/data.txt',
+            'foo-0.1/src/foo/py.typed',
             'foo-0.1/src/foo.egg-info',
             'foo-0.1/src/foo.egg-info/PKG-INFO',
             'foo-0.1/src/foo.egg-info/SOURCES.txt',
@@ -419,8 +423,10 @@ def test_build_with_pyproject_config(self, tmpdir, setup_script):
         }
         assert wheel_contents == {
             "foo/__init__.py",
+            "foo/__init__.pyi",  # include type information by default
             "foo/cli.py",
             "foo/data.txt",  # include_package_data defaults to True
+            "foo/py.typed",  # include type information by default
             "foo-0.1.dist-info/LICENSE.txt",
             "foo-0.1.dist-info/METADATA",
             "foo-0.1.dist-info/WHEEL",
diff --git a/setuptools/tests/test_build_py.py b/setuptools/tests/test_build_py.py
--- a/setuptools/tests/test_build_py.py
+++ b/setuptools/tests/test_build_py.py
@@ -319,3 +319,142 @@ def test_get_outputs(tmpdir_cwd):
         f"{build_lib}/mypkg/sub2/nested/__init__.py": "other/__init__.py",
         f"{build_lib}/mypkg/sub2/nested/mod3.py": "other/mod3.py",
     }
+
+
+class TestTypeInfoFiles:
+    PYPROJECTS = {
+        "default_pyproject": DALS(
+            """
+            [project]
+            name = "foo"
+            version = "1"
+            """
+        ),
+        "dont_include_package_data": DALS(
+            """
+            [project]
+            name = "foo"
+            version = "1"
+
+            [tools.setuptools]
+            include-package-data = false
+            """
+        ),
+        "exclude_type_info": DALS(
+            """
+            [project]
+            name = "foo"
+            version = "1"
+
+            [tools.setuptools]
+            include-package-data = false
+
+            [tool.setuptools.exclude-package-data]
+            "*" = ["py.typed", "*.pyi"]
+            """
+        ),
+    }
+
+    EXAMPLES = {
+        "simple_namespace": {
+            "directory_structure": {
+                "foo": {
+                    "bar.pyi": "",
+                    "py.typed": "",
+                    "__init__.py": "",
+                }
+            },
+            "expected_type_files": {"foo/bar.pyi", "foo/py.typed"},
+        },
+        "nested_inside_namespace": {
+            "directory_structure": {
+                "foo": {
+                    "bar": {
+                        "py.typed": "",
+                        "mod.pyi": "",
+                    }
+                }
+            },
+            "expected_type_files": {"foo/bar/mod.pyi", "foo/bar/py.typed"},
+        },
+        "namespace_nested_inside_regular": {
+            "directory_structure": {
+                "foo": {
+                    "namespace": {
+                        "foo.pyi": "",
+                    },
+                    "__init__.pyi": "",
+                    "py.typed": "",
+                }
+            },
+            "expected_type_files": {
+                "foo/namespace/foo.pyi",
+                "foo/__init__.pyi",
+                "foo/py.typed",
+            },
+        },
+    }
+
+    @pytest.mark.parametrize(
+        "pyproject", ["default_pyproject", "dont_include_package_data"]
+    )
+    @pytest.mark.parametrize("example", EXAMPLES.keys())
+    def test_type_files_included_by_default(self, tmpdir_cwd, pyproject, example):
+        structure = {
+            **self.EXAMPLES[example]["directory_structure"],
+            "pyproject.toml": self.PYPROJECTS[pyproject],
+        }
+        expected_type_files = self.EXAMPLES[example]["expected_type_files"]
+        jaraco.path.build(structure)
+
+        build_py = get_finalized_build_py()
+        outputs = get_outputs(build_py)
+        assert expected_type_files <= outputs
+
+    @pytest.mark.parametrize("pyproject", ["exclude_type_info"])
+    @pytest.mark.parametrize("example", EXAMPLES.keys())
+    def test_type_files_can_be_excluded(self, tmpdir_cwd, pyproject, example):
+        structure = {
+            **self.EXAMPLES[example]["directory_structure"],
+            "pyproject.toml": self.PYPROJECTS[pyproject],
+        }
+        expected_type_files = self.EXAMPLES[example]["expected_type_files"]
+        jaraco.path.build(structure)
+
+        build_py = get_finalized_build_py()
+        outputs = get_outputs(build_py)
+        assert expected_type_files.isdisjoint(outputs)
+
+    def test_stub_only_package(self, tmpdir_cwd):
+        structure = {
+            "pyproject.toml": DALS(
+                """
+                [project]
+                name = "foo-stubs"
+                version = "1"
+                """
+            ),
+            "foo-stubs": {"__init__.pyi": "", "bar.pyi": ""},
+        }
+        expected_type_files = {"foo-stubs/__init__.pyi", "foo-stubs/bar.pyi"}
+        jaraco.path.build(structure)
+
+        build_py = get_finalized_build_py()
+        outputs = get_outputs(build_py)
+        assert expected_type_files <= outputs
+
+
+def get_finalized_build_py(script_name="%build_py-test%"):
+    dist = Distribution({"script_name": script_name})
+    dist.parse_config_files()
+    build_py = dist.get_command_obj("build_py")
+    build_py.finalize_options()
+    return build_py
+
+
+def get_outputs(build_py):
+    build_dir = Path(build_py.build_lib)
+    return {
+        os.path.relpath(x, build_dir).replace(os.sep, "/")
+        for x in build_py.get_outputs()
+    }