Repository: python-poetry/poetry

Issue:
tool.poetry.packages handles wildcards * and ** incorrectly in pyproject.toml
- [x] I have searched the [issues](https://github.com/sdispater/poetry/issues) of this repo and believe that this is not a duplicate.

## Issue

The [documentation gives the following example](https://poetry.eustace.io/docs/pyproject/#packages) for using the `tool.poetry.packages` property of the `pyproject.toml` file:

```
[tool.poetry]
# ...
packages = [
    { include = "mypackage" },
    { include = "extra_package/**/*.py" },
]
```

However the command `poetry build` raises the following error when the` "extra_package/**/*.py"` path pattern matches multiple files/directories in at least one the of its path segments:

```
[ValueError]
/Users/maggyero/Desktop/project/extra_package/bar/baz.py is not a package.
```

with the following directory layout:

```
project
├── .git
├── mypackage
│   └── foo.py
├── extra_package
│   └── bar
│       ├── baz.py
│       └── qux.py
└── pyproject.toml
```

Here the last `*.py` path segment of the `"extra_package/**/*.py"` path pattern matches `baz.py` and `qux.py`, which triggers the `ValueError`. Deleting the `project/extra_package/bar/qux.py` file removes the `ValueError`. And adding a `project/extra_package/quux` directory raises the same `ValueError`, since this time the second `**` path segment of the `"extra_package/**/*.py"` path pattern matches `bar` and `qux`. So it seems that matching multiple files/directories in the path segments of a path pattern is not allowed. It looks like a _bug_, because each item of the `tool.poetry.packages` list is supposed to include a single top-level module or package and optionally specific submodules (like `"extra_package/**/*.py"` which includes the top-level `extra_package` package and submodules that have a `.py` extension).

Also, since the `tool.poetry.packages` property of the `pyproject.toml` file can include both directories and files, that is to say both packages and non-package modules, I think we should rename to `tool.poetry.modules`.


Discussion:
FYI @sdispater, @brycedrennan.
Hello,

that's the code that is responsible for this:

https://github.com/sdispater/poetry/blob/8fd63110b8de2479f3d65a86209ade65c788d96a/poetry/masonry/utils/package_include.py#L37-L63

So, whenever a glob is used, the poetry builder expect to find a `__init__.py` in the list of files after expanding the wildcards. But it doesn't expect one if only one specific folder or file is given.

To fix this issue, it is necessary to define the expected behavior. Is one allowed to include module files only? If a package should be included, is a `__init__.py` necessary (which isn't for python version >= 3.3)?

fin swimmer
@sdispater , @stephsamson: I would like to work on that, but need some guidance/decision about what should actually get included if the `packages` option is used.

Would be nice if you find some time to discuss this :)
@finswimmer thanks for volunteering to work on this issue :) I am already currently working on this ticket actually which is why this has been self-assigned but there are other bugs in the low priority lane [here](https://github.com/sdispater/poetry/projects/2) which need help!

But indeed you are right, the work necessary to be done is in `package_include` and `include`. Both of them have functional tests but no unit tests which may be one reason why this issue was not caught in the first place until reported.

Resolving patch:
diff --git a/poetry/masonry/builders/sdist.py b/poetry/masonry/builders/sdist.py
--- a/poetry/masonry/builders/sdist.py
+++ b/poetry/masonry/builders/sdist.py
@@ -237,7 +237,7 @@ def find_nearest_pkg(rel_path):
             if from_top_level == ".":
                 continue
 
-            is_subpkg = "__init__.py" in filenames
+            is_subpkg = any([filename.endswith(".py") for filename in filenames])
             if is_subpkg:
                 subpkg_paths.add(from_top_level)
                 parts = from_top_level.split(os.sep)
diff --git a/poetry/masonry/utils/package_include.py b/poetry/masonry/utils/package_include.py
--- a/poetry/masonry/utils/package_include.py
+++ b/poetry/masonry/utils/package_include.py
@@ -12,7 +12,6 @@ def __init__(self, base, include, formats=None, source=None):
             base = base / source
 
         super(PackageInclude, self).__init__(base, include, formats=formats)
-
         self.check_elements()
 
     @property
@@ -35,6 +34,8 @@ def refresh(self):  # type: () -> PackageInclude
         return self.check_elements()
 
     def check_elements(self):  # type: () -> PackageInclude
+        root = self._elements[0]
+
         if not self._elements:
             raise ValueError(
                 "{} does not contain any element".format(self._base / self._include)
@@ -44,20 +45,24 @@ def check_elements(self):  # type: () -> PackageInclude
             # Probably glob
             self._is_package = True
 
-            # The __init__.py file should be first
-            root = self._elements[0]
-            if root.name != "__init__.py":
-                raise ValueError("{} is not a package.".format(root))
+            # Packages no longer need an __init__.py in python3, but there must
+            # at least be one .py file for it to be considered a package
+            if not any([element.suffix == ".py" for element in self._elements]):
+                raise ValueError("{} is not a package.".format(root.name))
 
             self._package = root.parent.name
         else:
-            if self._elements[0].is_dir():
+            if root.is_dir():
                 # If it's a directory, we include everything inside it
-                self._package = self._elements[0].name
-                self._elements = sorted(list(self._elements[0].glob("**/*")))
+                self._package = root.name
+                self._elements = sorted(list(root.glob("**/*")))
+
+                if not any([element.suffix == ".py" for element in self._elements]):
+                    raise ValueError("{} is not a package.".format(root.name))
+
                 self._is_package = True
             else:
-                self._package = self._elements[0].stem
+                self._package = root.stem
                 self._is_module = True
 
         return self


Test patch:
diff --git a/tests/masonry/builders/fixtures/complete/my_package/sub_pkg3/foo.py b/tests/masonry/builders/fixtures/complete/my_package/sub_pkg3/foo.py
new file mode 100644
diff --git a/tests/masonry/builders/test_complete.py b/tests/masonry/builders/test_complete.py
--- a/tests/masonry/builders/test_complete.py
+++ b/tests/masonry/builders/test_complete.py
@@ -265,6 +265,7 @@ def test_complete_no_vcs():
         "my_package/sub_pkg1/__init__.py",
         "my_package/sub_pkg2/__init__.py",
         "my_package/sub_pkg2/data2/data.json",
+        "my_package/sub_pkg3/foo.py",
         "my_package-1.2.3.dist-info/entry_points.txt",
         "my_package-1.2.3.dist-info/LICENSE",
         "my_package-1.2.3.dist-info/WHEEL",
diff --git a/tests/masonry/builders/test_sdist.py b/tests/masonry/builders/test_sdist.py
--- a/tests/masonry/builders/test_sdist.py
+++ b/tests/masonry/builders/test_sdist.py
@@ -124,6 +124,7 @@ def test_make_setup():
         "my_package",
         "my_package.sub_pkg1",
         "my_package.sub_pkg2",
+        "my_package.sub_pkg3",
     ]
     assert ns["install_requires"] == ["cachy[msgpack]>=0.2.0,<0.3.0", "cleo>=0.6,<0.7"]
     assert ns["entry_points"] == {
@@ -178,6 +179,7 @@ def test_find_files_to_add():
             Path("my_package/sub_pkg1/__init__.py"),
             Path("my_package/sub_pkg2/__init__.py"),
             Path("my_package/sub_pkg2/data2/data.json"),
+            Path("my_package/sub_pkg3/foo.py"),
             Path("pyproject.toml"),
         ]
     )
@@ -213,7 +215,12 @@ def test_find_packages():
     pkg_dir, packages, pkg_data = builder.find_packages(include)
 
     assert pkg_dir is None
-    assert packages == ["my_package", "my_package.sub_pkg1", "my_package.sub_pkg2"]
+    assert packages == [
+        "my_package",
+        "my_package.sub_pkg1",
+        "my_package.sub_pkg2",
+        "my_package.sub_pkg3",
+    ]
     assert pkg_data == {
         "": ["*"],
         "my_package": ["data1/*"],
diff --git a/tests/masonry/utils/__init__.py b/tests/masonry/utils/__init__.py
new file mode 100644
diff --git a/tests/masonry/utils/fixtures/with_includes/__init__.py b/tests/masonry/utils/fixtures/with_includes/__init__.py
new file mode 100644
diff --git a/tests/masonry/utils/fixtures/with_includes/bar/baz.py b/tests/masonry/utils/fixtures/with_includes/bar/baz.py
new file mode 100644
diff --git a/tests/masonry/utils/fixtures/with_includes/extra_package/some_dir/foo.py b/tests/masonry/utils/fixtures/with_includes/extra_package/some_dir/foo.py
new file mode 100644
diff --git a/tests/masonry/utils/fixtures/with_includes/extra_package/some_dir/quux.py b/tests/masonry/utils/fixtures/with_includes/extra_package/some_dir/quux.py
new file mode 100644
diff --git a/tests/masonry/utils/fixtures/with_includes/not_a_python_pkg/baz.txt b/tests/masonry/utils/fixtures/with_includes/not_a_python_pkg/baz.txt
new file mode 100644
diff --git a/tests/masonry/utils/test_package_include.py b/tests/masonry/utils/test_package_include.py
new file mode 100644
--- /dev/null
+++ b/tests/masonry/utils/test_package_include.py
@@ -0,0 +1,43 @@
+import pytest
+
+from poetry.masonry.utils.package_include import PackageInclude
+from poetry.utils._compat import Path
+
+
+fixtures_dir = Path(__file__).parent / "fixtures"
+with_includes = fixtures_dir / "with_includes"
+
+
+def test_package_include_with_multiple_dirs():
+    pkg_include = PackageInclude(base=fixtures_dir, include="with_includes")
+    assert pkg_include.elements == [
+        with_includes / "__init__.py",
+        with_includes / "bar",
+        with_includes / "bar/baz.py",
+        with_includes / "extra_package",
+        with_includes / "extra_package/some_dir",
+        with_includes / "extra_package/some_dir/foo.py",
+        with_includes / "extra_package/some_dir/quux.py",
+        with_includes / "not_a_python_pkg",
+        with_includes / "not_a_python_pkg/baz.txt",
+    ]
+
+
+def test_package_include_with_simple_dir():
+    pkg_include = PackageInclude(base=with_includes, include="bar")
+    assert pkg_include.elements == [with_includes / "bar/baz.py"]
+
+
+def test_package_include_with_nested_dir():
+    pkg_include = PackageInclude(base=with_includes, include="extra_package/**/*.py")
+    assert pkg_include.elements == [
+        with_includes / "extra_package/some_dir/foo.py",
+        with_includes / "extra_package/some_dir/quux.py",
+    ]
+
+
+def test_package_include_with_no_python_files_in_dir():
+    with pytest.raises(ValueError) as e:
+        PackageInclude(base=with_includes, include="not_a_python_pkg")
+
+    assert str(e.value) == "not_a_python_pkg is not a package."