Repository: capitalone/edgetest (language: python)

Issue:
Better support for extras in the upgrade portion
Right now if you want to upgrade a package which has `mypackage[extras]` you need to update it as `mypackage`. I think we can provide better support for the `extras` to align better as there could be cases where the items in `extras` are not being bumped properly.

@ak-gupta thoughts?
Better support for extras in the upgrade portion
Right now if you want to upgrade a package which has `mypackage[extras]` you need to update it as `mypackage`. I think we can provide better support for the `extras` to align better as there could be cases where the items in `extras` are not being bumped properly.

@ak-gupta thoughts?


Discussion / hints:
@ak-gupta So I'm thinking we can add something like the following here: https://github.com/capitalone/edgetest/blob/dev/edgetest/core.py#L152

```python
pattern = "\[.+\]"
upgrade_wo_extras = [re.sub(pattern, "", pkg) for pkg in self.upgrade]
return [pkg for pkg in outjson if pkg.get("name", "") in upgrade_wo_extras]
```

In the current code something like `dask[dataframe]` does not pass the conditional in the list comprehension. 

```python
return [pkg for pkg in outjson if pkg.get("name", "") in self.upgrade]
```
It works out to something like this:

```python
In [1]: upgrade = ['dask[dataframe]']

In [2]: pkg_from_pip = {'name': 'dask', 'version': '2022.3.0'}

In [3]: pkg_from_pip.get("name", "") in upgrade
Out[3]: False
```
Yeah that should work. My first thought it `pkg.split("[")[0]` could be simpler but either should fix the problem
We should also validate that the upgrading doesn't nuke the extra from the requirements. From looking at the `pkg_resources.parse_requirements` function it _shouldn't_, but maybe we add an extra to one of the upgrade packages in the integration testing to make sure
I'm all for simpler code and one less import. :)
@ak-gupta So I'm thinking we can add something like the following here: https://github.com/capitalone/edgetest/blob/dev/edgetest/core.py#L152

```python
pattern = "\[.+\]"
upgrade_wo_extras = [re.sub(pattern, "", pkg) for pkg in self.upgrade]
return [pkg for pkg in outjson if pkg.get("name", "") in upgrade_wo_extras]
```

In the current code something like `dask[dataframe]` does not pass the conditional in the list comprehension. 

```python
return [pkg for pkg in outjson if pkg.get("name", "") in self.upgrade]
```
It works out to something like this:

```python
In [1]: upgrade = ['dask[dataframe]']

In [2]: pkg_from_pip = {'name': 'dask', 'version': '2022.3.0'}

In [3]: pkg_from_pip.get("name", "") in upgrade
Out[3]: False
```
Yeah that should work. My first thought it `pkg.split("[")[0]` could be simpler but either should fix the problem
We should also validate that the upgrading doesn't nuke the extra from the requirements. From looking at the `pkg_resources.parse_requirements` function it _shouldn't_, but maybe we add an extra to one of the upgrade packages in the integration testing to make sure
I'm all for simpler code and one less import. :)

Fix patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index c7ba5e7..a241b71 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -5,7 +5,7 @@ repos:
     hooks:
       - id: black
         types: [file, python]
-        language_version: python3.7
+        language_version: python3.9
   - repo: https://github.com/pre-commit/mirrors-isort
     rev: v5.5.4
     hooks:
@@ -16,7 +16,7 @@ repos:
       - id: flake8
         additional_dependencies: [flake8-docstrings]
         files: ^edgetest/
-        language_version: python3.7
+        language_version: python3.9
   - repo: https://github.com/jazzband/pip-tools
     rev: 5.5.0
     hooks:
diff --git a/docs/source/conf.py b/docs/source/conf.py
index 84a95ed..e74a40b 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -25,7 +25,7 @@
 author = "Akshay Gupta"
 
 # The short X.Y version
-version = "2022.3.0"
+version = "2022.3.1"
 # The full version, including alpha/beta/rc tags
 release = ""
 
diff --git a/setup.cfg b/setup.cfg
index dd968f8..b8a294e 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -10,13 +10,13 @@ license = Apache Software License
 maintainer = Akshay Gupta
 maintainer_email = akshay.gupta2@capitalone.com
 url = https://github.com/capitalone/edgetest
-python_requires = 
+python_requires =
 	>=3.7.0
-project_urls = 
+project_urls =
 	Documentation = https://capitalone.github.io/edgetest/
 	Bug Tracker = https://github.com/capitalone/edgetest/issues
 	Source Code = https://github.com/capitalone/edgetest
-classifiers = 
+classifiers =
 	Intended Audience :: Developers
 	Natural Language :: English
 	Operating System :: OS Independent
@@ -31,7 +31,7 @@ classifiers =
 zip_safe = False
 include_package_data = True
 packages = find:
-install_requires = 
+install_requires =
 	Cerberus<=1.3.4,>=1.3.0
 	click<=8.0.4,>=7.0
 	pluggy<=1.0.0,>=1.0.0
@@ -39,21 +39,21 @@ install_requires =
 	packaging<=21.3,>20.6
 
 [options.extras_require]
-docs = 
+docs =
 	furo
 	sphinx
 	sphinx-copybutton
 	sphinx-tabs
 	pillow
 	recommonmark
-tests = 
+tests =
 	coverage
 	flake8
 	mypy
 	pydocstyle
 	pytest
 	pytest-cov
-qa = 
+qa =
 	pylint
 	pip-tools
 	pre-commit
@@ -62,12 +62,12 @@ qa =
 	types-click
 	types-pkg_resources
 	types-tabulate
-build = 
+build =
 	build
 	twine
 	wheel
 	bumpver
-dev = 
+dev =
 	coverage
 	flake8
 	mypy
@@ -94,21 +94,21 @@ dev =
 	bumpver
 
 [options.entry_points]
-console_scripts = 
+console_scripts =
 	edgetest = edgetest.interface:cli
 
 [bumpver]
-current_version = "2022.3.0"
+current_version = "2022.3.1"
 version_pattern = "YYYY.MM.INC0"
 commit_message = "Bump {old_version} to {new_version}"
 commit = True
 
 [bumpver:file_patterns]
-docs/source/conf.py = 
+docs/source/conf.py =
 	version = "{version}"
-setup.cfg = 
+setup.cfg =
 	current_version = "{version}"
-edgetest/__init__.py = 
+edgetest/__init__.py =
 	__version__ = "{version}"
 
 [aliases]
@@ -141,21 +141,20 @@ allow_redefinition = True
 pylint_minimum_score = 9.5
 
 [tool:pytest]
-markers = 
+markers =
 	unit: mark unit tests that do not require external interfaces and use mocking
 	integration: mark test that interact with an external system
 addopts = --verbose
 
 [edgetest.envs.core]
 python_version = 3.9
-upgrade = 
+upgrade =
 	Cerberus
 	click
 	pluggy
 	tabulate
 	packaging
-extras = 
+extras =
 	tests
-deps = 
+deps =
 	pip-tools
-


Test patch:
diff --git a/.github/workflows/test-package.yml b/.github/workflows/test-package.yml
index 5177a0c..c6e4420 100644
--- a/.github/workflows/test-package.yml
+++ b/.github/workflows/test-package.yml
@@ -1,5 +1,6 @@
 # Install and test
 name: Python package
+
 on:
     push:
         branches:
diff --git a/edgetest/__init__.py b/edgetest/__init__.py
index 8d67082..62518e4 100644
--- a/edgetest/__init__.py
+++ b/edgetest/__init__.py
@@ -1,7 +1,7 @@
 """Package initialization."""
 
 
-__version__ = "2022.3.0"
+__version__ = "2022.3.1"
 
 __title__ = "edgetest"
 __description__ = "Bleeding edge dependency testing"
diff --git a/edgetest/core.py b/edgetest/core.py
index 576672b..0e44d1c 100644
--- a/edgetest/core.py
+++ b/edgetest/core.py
@@ -150,7 +150,8 @@ def upgraded_packages(self) -> List[Dict[str, str]]:
         out, _ = _run_command(self.python_path, "-m", "pip", "list", "--format", "json")
         outjson = json.loads(out)
 
-        return [pkg for pkg in outjson if pkg.get("name", "") in self.upgrade]
+        upgrade_wo_extras = [pkg.split("[")[0] for pkg in self.upgrade]
+        return [pkg for pkg in outjson if pkg.get("name", "") in upgrade_wo_extras]
 
     def run_tests(self, command: str) -> int:
         """Run the tests in the package directory.
diff --git a/tests/test_integration.py b/tests/test_integration.py
index 2b8d968..dd6b6ab 100644
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -1,5 +1,6 @@
 """Create a fake package and test."""
-
+import configparser
+import sys
 from pathlib import Path
 
 import pytest
@@ -31,6 +32,36 @@
     tests
 """
 
+
+SETUP_CFG_DASK = """
+[metadata]
+name = toy_package
+version = 0.1.0
+description = Fake description
+python_requires =
+    >=3.7.0
+
+[options]
+zip_safe = False
+include_package_data = True
+packages = find:
+install_requires =
+    click<=8.0.0,>=7.1
+    dask[dataframe]<=2022.1.0
+
+[options.extras_require]
+tests =
+    pytest
+
+[edgetest.envs.core]
+extras =
+    tests
+upgrade =
+    click
+    dask[dataframe]
+"""
+
+
 SETUP_PY = """
 from setuptools import setup
 
@@ -49,7 +80,10 @@ def test_func():
     main()
 """
 
+PY_VER = f"python{sys.version_info.major}.{sys.version_info.minor}"
+
 
+@pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
 @pytest.mark.integration
 def test_toy_package():
     """Test using edgetest with a toy package."""
@@ -75,3 +109,54 @@ def test_toy_package():
         assert result.exit_code == 0
         assert Path(loc, ".edgetest").is_dir()
         assert Path(loc, ".edgetest", "pandas").is_dir()
+
+        if not sys.platform == "win32":
+            assert Path(
+                loc, ".edgetest", "pandas", "lib", PY_VER, "site-packages", "pandas"
+            ).is_dir()
+
+
+@pytest.mark.integration
+def test_toy_package_dask():
+    """Test using edgetest with a toy package."""
+    runner = CliRunner()
+
+    with runner.isolated_filesystem() as loc:
+        with open("setup.cfg", "w") as outfile:
+            outfile.write(SETUP_CFG_DASK)
+        with open("setup.py", "w") as outfile:
+            outfile.write(SETUP_PY)
+        # Make a directory for the module
+        Path(loc, "toy_package").mkdir()
+        with open(Path(loc, "toy_package", "__init__.py"), "w") as outfile:
+            outfile.write(MODULE_CODE)
+        # Make a directory for the tests
+        Path(loc, "tests").mkdir()
+        with open(Path(loc, "tests", "test_main.py"), "w") as outfile:
+            outfile.write(TEST_CODE)
+
+        # Run the CLI
+        result = runner.invoke(cli, ["--config=setup.cfg", "--export"])
+
+        assert result.exit_code == 0
+        assert Path(loc, ".edgetest").is_dir()
+        assert Path(loc, ".edgetest", "core").is_dir()
+
+        if not sys.platform == "win32":
+            assert Path(
+                loc, ".edgetest", "core", "lib", PY_VER, "site-packages", "dask"
+            ).is_dir()
+            assert Path(
+                loc, ".edgetest", "core", "lib", PY_VER, "site-packages", "pandas"
+            ).is_dir()
+            assert Path(
+                loc, ".edgetest", "core", "lib", PY_VER, "site-packages", "click"
+            ).is_dir()
+
+        config = configparser.ConfigParser()
+        config.read("setup.cfg")
+
+        assert "click" in config["options"]["install_requires"]
+        assert "dask[dataframe]" in config["options"]["install_requires"]
+        assert config["edgetest.envs.core"]["extras"] == "\ntests"
+        assert config["edgetest.envs.core"]["upgrade"] == "\nclick\ndask[dataframe]"