Repository: fredrikaverpil/creosote (language: python)

Issue:
Allow configuration via pyproject.toml
### What feature request would you like to suggest?

It would be nice if we could configure creosote via a `[tool.creosote]` section in `pyproject.toml`. Similar to other tools like pytest, CLI arguments could override values provided in `pyproject.toml`.

E.g.
```
[tool.creosote]
venv=".venv"
paths="my_project/"
deps-file="pyproject.toml"
sections=tool.poetry.dependencies
exclude-deps=[
    "importlib_resources",
    "pydantic",
]
```

Implementation details:
* Care must be taken to support lists for arguments that can be supplied multiple times.

### Will the feature request break backwards compatibility?

Yes, since all projects would currently have a non-existant `[tool.creosote]` section, so then the default behavior is current behavior.


Discussion / hints:
Thanks for this suggestion. I'm likely not going to implement this myself, but I'm open to review/accept PRs on this. Good timing @BrianPugh, just in time for [hacktoberfest](https://hacktoberfest.com) 2023 😉 

I recommend implementation for this breaks backwards compatibility and targets a 3.0.0 release. There are some changes in regards to how arguments are passed in (see `v3-args` under [features](https://github.com/fredrikaverpil/creosote#%EF%B8%8F-features)) that should become default with 3.0.0 too. So it might be a good idea to make `v3-args` the default and then build from there as part of this work.

### Steps to get started

For anyone who is interested in picking this up: the [`cli.py`](https://github.com/fredrikaverpil/creosote/blob/main/src/creosote/cli.py) file contains all the code you need to modify. I'd be happy to schedule a call or a chat to plan the work or provide guidance if it is helpful. Steps to get started:

1. Make `v3-args` into the default behavior by removing the `ARGS_CAN_BE_SPECIFIED_MULTIPLE_TIMES` feature and update the associated if-condition.
2. Store hard-coded variables which hold today's default values for each CLI argument. See each `default=...` for each argparse argument.
3. Optionally cater for being able to auto-detect the activated virtual environment (see #168), likely in step 2.
4. Before parsing the CLI arguments (today done in `parse_args()`), read the `pyproject.toml`'s `[tool.creosote]` section. You can use the [`toml`](https://github.com/uiri/toml) and  [`dotty-dict`](https://github.com/pawelzny/dotty_dict) dependencies here if it helps to parse it, since these are already project dependencies. If values are found, use these instead of the hard-coded default values mentioned in step 2.
5. Finally, set each argument's `default` to either the hard-coded value or the one your read from `pyproject.toml`.

Then there will also be tests that will break following these changes, of course, that has to be updated.

Fix patch:
diff --git a/src/creosote/__main__.py b/src/creosote/__main__.py
new file mode 100644
index 0000000..a0d6565
--- /dev/null
+++ b/src/creosote/__main__.py
@@ -0,0 +1,5 @@
+import sys
+
+from creosote.cli import main
+
+sys.exit(main(sys.argv[1:]))
diff --git a/src/creosote/cli.py b/src/creosote/cli.py
index b586699..5b9f295 100644
--- a/src/creosote/cli.py
+++ b/src/creosote/cli.py
@@ -1,179 +1,21 @@
-import argparse
-import glob
 import sys
-from enum import Enum
 
 from loguru import logger
 
 from creosote import formatters, parsers, resolvers
 from creosote.__about__ import __version__
-
-
-class Features(Enum):
-    """Features that can be enabled via the --use-feature flag."""
-
-    FAIL_EXCLUDED_AND_NOT_INSTALLED = "fail-excluded-and-not-installed"
-    ARGS_CAN_BE_SPECIFIED_MULTIPLE_TIMES = "v3-args"
-
-
-class CustomAppendAction(argparse.Action):
-    """Custom action to append values to a list.
-
-    When using the `append` action, the default value is not removed
-    from the list. This problem is described in
-    https://github.com/python/cpython/issues/60603
-
-    This custom action aims to fix this problem by removing the default
-    value when the argument is specified for the first time.
-    """
-
-    def __init__(self, option_strings, dest, nargs=None, **kwargs):
-        """Initialize the action."""
-        self.called_times = 0
-        self.default_value = kwargs.get("default")
-        super().__init__(option_strings, dest, **kwargs)
-
-    def __call__(self, parser, namespace, values, option_string=None):
-        """When the argument is specified on the commandline."""
-        current_values = getattr(namespace, self.dest)
-
-        if self.called_times == 0 and current_values == self.default_value:
-            current_values = []
-
-        current_values.append(values)
-        setattr(namespace, self.dest, current_values)
-        self.called_times += 1
-
-
-def parse_args(args):
-    parser = argparse.ArgumentParser(
-        description=(
-            "Prevent bloated virtual environments by identifing installed, "
-            "but unused, dependencies"
-        ),
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
-    )
-    parser.add_argument(
-        "-V",
-        "--version",
-        dest="version",
-        action="version",
-        version=__version__,
-        help="show version and exit",
-    )
-    parser.add_argument(
-        "--verbose",
-        dest="verbose",
-        action="store_true",
-        help="increase output verbosity",
-    )
-    parser.add_argument(
-        "-f",
-        "--format",
-        dest="format",
-        default="default",
-        choices=["default", "no-color", "porcelain"],
-        help="output format",
-    )
-
-    if Features.ARGS_CAN_BE_SPECIFIED_MULTIPLE_TIMES.value in sys.argv:
-        # v3.0 behavior
-        parser.add_argument(
-            "-p",
-            "--path",
-            dest="paths",
-            metavar="PATH",
-            action=CustomAppendAction,
-            default=["src"],
-            help="path(s) to Python source code to scan for imports",
-        )
-        parser.add_argument(
-            "-s",
-            "--section",
-            dest="sections",
-            metavar="TOML_SECTION",
-            action=CustomAppendAction,
-            default=["project.dependencies"],
-            help="pyproject.toml section(s) to scan for dependencies",
-        )
-        parser.add_argument(
-            "--exclude-dep",
-            dest="exclude_deps",
-            metavar="DEPENDENCY",
-            action="append",
-            default=[],
-            help="dependency(ies) to exclude from the scan",
-        )
-    else:
-        # v2.x behavior
-        parser.add_argument(
-            "-p",
-            "--paths",
-            dest="paths",
-            default=glob.glob("src"),
-            nargs="*",
-            help="paths(s) to Python source code to scan for imports",
-        )
-        parser.add_argument(
-            "-s",
-            "--sections",
-            dest="sections",
-            metavar="TOML_SECTION",
-            nargs="*",
-            default=["project.dependencies"],
-            help="pyproject.toml section(s) to scan for dependencies",
-        )
-        parser.add_argument(
-            "--exclude-deps",
-            dest="exclude_deps",
-            metavar="DEPENDENCY",
-            nargs="*",
-            default=[],
-            help="dependencies to exclude from the scan",
-        )
-
-    parser.add_argument(
-        "-d",
-        "--deps-file",
-        dest="deps_file",
-        metavar="PATH",
-        default="pyproject.toml",
-        help="path to the pyproject.toml or requirements[.txt|.in] file",
-    )
-    parser.add_argument(
-        "-v",
-        "--venv",
-        dest="venvs",
-        metavar="PATH",
-        action=CustomAppendAction,
-        default=[".venv"],
-        help="path(s) to the virtual environment (or site-packages)",
-    )
-    parser.add_argument(
-        "--use-feature",
-        dest="features",
-        metavar="FEATURE",
-        action="append",
-        choices=[v.value for v in Features.__members__.values()],
-        default=[],
-        help=(
-            "enable new/experimental functionality, "
-            "that may be backward incompatible"
-        ),
-    )
-
-    parsed_args = parser.parse_args(args)
-
-    return parsed_args
+from creosote.config import Features, parse_args
 
 
 def main(args_=None):
-    args = parse_args(args_)
-
+    args, default_config = parse_args(args_)
     formatters.configure_logger(verbose=args.verbose, format_=args.format)
 
     logger.debug(f"Creosote version: {__version__}")
     logger.debug(f"Command: creosote {' '.join(sys.argv[1:])}")
+    logger.debug(
+        f"Default configuration (may have loaded pyproject.toml): {default_config}"
+    )
     logger.debug(f"Arguments: {args}")
 
     if args.features:
diff --git a/src/creosote/config.py b/src/creosote/config.py
new file mode 100644
index 0000000..5414151
--- /dev/null
+++ b/src/creosote/config.py
@@ -0,0 +1,167 @@
+import argparse
+import dataclasses
+import os
+import typing
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import List, Literal
+
+import toml
+
+from creosote.__about__ import __version__
+
+
+@dataclass
+class Config:
+    """Structured configuration data.
+
+    It is important that these attributes exactly match
+    the ``dest`` specified in ``add_argument``.
+    """
+
+    format: Literal["default", "no-color", "porcelain"] = "default"  # noqa: A003
+    paths: List[str] = field(default_factory=lambda: ["src"])
+    sections: List[str] = field(default_factory=lambda: ["project.dependencies"])
+    exclude_deps: List[str] = field(default_factory=list)
+    deps_file: str = "pyproject.toml"
+    venvs: List[str] = field(
+        default_factory=lambda: [os.environ.get("VIRTUAL_ENV", ".venv")]
+    )
+    features: List[str] = field(default_factory=list)
+
+
+class Features(Enum):
+    """Features that can be enabled via the --use-feature flag."""
+
+    FAIL_EXCLUDED_AND_NOT_INSTALLED = "fail-excluded-and-not-installed"
+
+
+class CustomAppendAction(argparse.Action):
+    """Custom action to append values to a list.
+
+    When using the `append` action, the default value is not removed
+    from the list. This problem is described in
+    https://github.com/python/cpython/issues/60603
+
+    This custom action aims to fix this problem by removing the default
+    value when the argument is specified for the first time.
+    """
+
+    def __init__(self, option_strings, dest, nargs=None, **kwargs):
+        """Initialize the action."""
+        self.called_times = 0
+        self.default_value = kwargs.get("default")
+        super().__init__(option_strings, dest, **kwargs)
+
+    def __call__(self, parser, namespace, values, option_string=None):
+        """When the argument is specified on the commandline."""
+        current_values = getattr(namespace, self.dest)
+
+        if self.called_times == 0 and current_values == self.default_value:
+            current_values = []
+
+        current_values.append(values)
+        setattr(namespace, self.dest, current_values)
+        self.called_times += 1
+
+
+def parse_args(args):
+    parser = argparse.ArgumentParser(
+        description=(
+            "Prevent bloated virtual environments by identifing installed, "
+            "but unused, dependencies"
+        ),
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    parser.add_argument(
+        "--verbose",
+        dest="verbose",
+        action="store_true",
+        help="increase output verbosity",
+    )
+    parser.add_argument(
+        "-f",
+        "--format",
+        dest="format",
+        choices=typing.get_args(Config.__annotations__["format"]),
+        help="output format",
+    )
+    parser.add_argument(
+        "-V",
+        "--version",
+        dest="version",
+        action="version",
+        version=__version__,
+        help="show version and exit",
+    )
+    parser.add_argument(
+        "-p",
+        "--path",
+        dest="paths",
+        metavar="PATH",
+        action=CustomAppendAction,  # TODO: add tests for v3-args before releasing 3.0.0
+        help="path(s) to Python source code to scan for imports",
+    )
+    parser.add_argument(
+        "-s",
+        "--section",
+        dest="sections",
+        metavar="TOML_SECTION",
+        action=CustomAppendAction,  # TODO: add tests for v3-args before releasing 3.0.0
+        help="pyproject.toml section(s) to scan for dependencies",
+    )
+    parser.add_argument(
+        "--exclude-dep",
+        dest="exclude_deps",
+        metavar="DEPENDENCY",
+        action="append",  # TODO: add tests for v3-args before releasing 3.0.0
+        help="dependency(ies) to exclude from the scan",
+    )
+    parser.add_argument(
+        "-d",
+        "--deps-file",
+        dest="deps_file",
+        metavar="PATH",
+        help="path to the pyproject.toml or requirements[.txt|.in] file",
+    )
+    parser.add_argument(
+        "-v",
+        "--venv",
+        dest="venvs",
+        metavar="PATH",
+        action=CustomAppendAction,
+        help="path(s) to the virtual environment (or site-packages)",
+    )
+    parser.add_argument(
+        "--use-feature",
+        dest="features",
+        metavar="FEATURE",
+        action="append",
+        choices=[v.value for v in Features.__members__.values()],
+        help=(
+            "enable new/experimental functionality, "
+            "that may be backward incompatible"
+        ),
+    )
+
+    defaults = load_defaults()
+    parser.set_defaults(**dataclasses.asdict(defaults))
+    parsed_args = parser.parse_args(args)
+    return parsed_args, defaults
+
+
+def load_defaults(src: str = "pyproject.toml") -> Config:
+    """Load pyproject.toml defaults form user config.
+
+    Expects user configuration at ``[tool.creosote]``.
+    """
+
+    try:
+        with open(src, "r", encoding="utf-8") as f:
+            project_config = toml.loads(f.read())
+    except FileNotFoundError:
+        project_config = {}
+    creosote_config = project_config.get("tool", {}).get("creosote", {})
+    # Convert all hyphens to underscores
+    creosote_config = {k.replace("-", "_"): v for k, v in creosote_config.items()}
+    return Config(**creosote_config)


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 1fd2df8..4afdac1 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -1,10 +1,11 @@
+import os
 from pathlib import Path
 from typing import Callable, List, Tuple
 
 import pytest
 from _pytest.capture import CaptureFixture
 
-from creosote import cli
+from creosote import cli, config
 
 
 def test_no_unused_and_found_in_top_level_txt(
@@ -111,7 +112,6 @@ def test_detected_indirectly_used_but_not_imported_and_excluded(
     """Excluded dependency is used but never imported by source code."""
     mock_dependencies_from_pyproject_toml(dependency_names=["hello", "bye"])
     mock_imports_from_source_code(import_names=["hello"])
-    excluded_dependencies = "bye"
 
     venv_path, _top_level_txt_path = venv_with_top_level_txt(
         dependency_name="hello", contents=["hello"]
@@ -123,8 +123,8 @@ def test_detected_indirectly_used_but_not_imported_and_excluded(
         [
             "--venv",
             str(venv_path),
-            "--exclude-deps",
-            excluded_dependencies,
+            "--exclude-dep",
+            "bye",
             "--format",
             "porcelain",
         ]
@@ -154,7 +154,6 @@ def test_unused_found_because_excluded_but_not_installed(  # noqa: PLR0913
     """Excluded dependency is used but never imported by source code."""
     mock_dependencies_from_pyproject_toml(dependency_names=["hello", "bye"])
     mock_imports_from_source_code(import_names=["hello"])
-    excluded_dependencies = "bye"
 
     venv_path, _site_packages_path = create_venv
     expected_unused_packages: List[str] = []
@@ -162,8 +161,8 @@ def test_unused_found_because_excluded_but_not_installed(  # noqa: PLR0913
     args = [
         "--venv",
         str(venv_path),
-        "--exclude-deps",
-        excluded_dependencies,
+        "--exclude-dep",
+        "bye",
         "--format",
         "porcelain",
     ]
@@ -175,3 +174,70 @@ def test_unused_found_because_excluded_but_not_installed(  # noqa: PLR0913
 
     assert captured.out.splitlines() == expected_unused_packages
     assert exit_code == expected_exit_code
+
+
+def test_load_defaults_no_file(tmp_path):
+    pyproject = tmp_path / "pyproject.toml"
+    configuration = config.load_defaults(pyproject)
+    assert configuration == config.Config()  # Should return a default config
+
+
+def test_load_defaults_no_tool_section(tmp_path):
+    pyproject = tmp_path / "pyproject.toml"
+    pyproject.write_text("[foo]")
+    configuration = config.load_defaults(pyproject)
+    assert configuration == config.Config()  # Should return a default config
+
+
+def test_load_defaults_no_tool_creosote_section(tmp_path):
+    pyproject = tmp_path / "pyproject.toml"
+    pyproject.write_text("[tool.foo]")
+    configuration = config.load_defaults(pyproject)
+    assert configuration == config.Config()  # Should return a default config
+
+
+def test_load_defaults_tool_creosote_section_simple(tmp_path):
+    pyproject = tmp_path / "pyproject.toml"
+    pyproject.write_text('[tool.creosote]\nvenvs=["foo"]')
+    configuration = config.load_defaults(pyproject)
+    assert configuration == config.Config(venvs=["foo"])
+
+
+def test_load_defaults_tool_creosote_section_complex(tmp_path):
+    """More close to a real configuration."""
+    pyproject = tmp_path / "pyproject.toml"
+    pyproject.write_text(
+        """
+        [tool.creosote]
+        venvs=[".virtual_environment"]
+        paths=["src"]
+        deps-file="requirements.txt"
+        exclude-deps=[
+            "importlib_resources",
+            "pydantic",
+        ]
+    """
+    )
+    configuration = config.load_defaults(pyproject)
+    assert configuration == config.Config(
+        venvs=[".virtual_environment"],
+        paths=["src"],
+        deps_file="requirements.txt",
+        exclude_deps=[
+            "importlib_resources",
+            "pydantic",
+        ],  # Tests hyphen -> underscore logic.
+    )
+
+
+def test_load_defaults_no_venv():
+    # Unset VIRTUAL_ENV environment variable
+    os.environ.pop("VIRTUAL_ENV", None)
+    configuration = config.Config()
+    assert configuration.venvs == [".venv"]
+
+
+def test_load_defaults_set_venv():
+    os.environ["VIRTUAL_ENV"] = "foo"
+    configuration = config.Config()
+    assert configuration.venvs == ["foo"]