Repository: pypa/setuptools

Issue:
extras_require in setup.cfg doesn't allow hyphen in extras name
I'm porting a `setup.py` file to declarative `setup.cfg`. In this package, an extra [contains a hyphen](https://github.com/pypa/twine/blob/6c4d5ecf2596c72b89b969ccc37b82c160645df8/setup.py#L91). This works when defined in `setup.py` but does not when defined in `setup.cfg`. If I change the hyphen to an underscore, it works again, but this would break the extra for users.

To reproduce:

`setup.py`:

```py
from setuptools import setup

setup()
```

`setup.cfg`:

```ini
[options.extras_require]
extra-a =
    pytest
extra_b =
    pytest
```

`tox.ini` (to automate the virtual env creation)

```ini
[testenv:a]
extras = extra-a

[testenv:b]
extras = extra_b
```

Then run:

```
$ tox -e a
$ tox -e b
```

Notice the extras are installed in the "b" environment but not the "a" environment. AFAICT, the only difference is the use of a hyphen instead of an underscore.


Discussion:
It looks like distutils automatically converts hyphens to underscores internally when reading `setup.cfg`:

https://github.com/python/cpython/blob/4aa917c5feaec07a6f6db87b34185ab6180e20ee/Lib/distutils/dist.py#L414
@jdufresne According to [this guide](https://packaging.python.org/guides/distributing-packages-using-setuptools/#name), underscores, hyphens and periods are all supposed to be treated equivalently.when normalizing names, referencing [PEP 508](https://www.python.org/dev/peps/pep-0508/), but I don't see anything about that there or in [PEP 423](https://www.python.org/dev/peps/pep-0423/).

Presumably if `extra-a` and `extra_a` are actually equivalent, this should not be an issue, right? If that's the case, why are you seeing a difference in behavior?
Thanks for taking a look. In that case, do you think this is an issue in pip and not setuptools? If I run the command:

```
pip install .[extra-a]
```

The extras are not installed. But if I run the command

```
pip install .[extra_a]
```

The extras are installed.

Should I file this issue with pip?
I'm not quite sure if there's an authoritative claim that those should be treated the same or if it was describing the behavior of a specific tool.

It could be that pip is not normalizing correctly or it could be that we should be normalizing to `extra_a`, or that none of us should be doing any normalizing at all.

We'll at least leave this ticket open until we get to the bottom of this. Presumably a change needs to be made in at least one of: setuptools, pip, the packaging tutorials.
> According to this guide, underscores, hyphens and periods are all supposed to be treated equivalently when normalizing names

When I read that reference, I see explicit mention of "project" names, but not necessarily other names. 

I'm strongly in favor of honoring the user's intention here and if for nothing other than feature parity, support untransformed names for extras from declarative config.
That distutils code isn't readily tweakable. I think the preferred solution is for setuptools not to rely on distutils to parse the options.
Ugh. The `setuptools.config` implementation is really intertwined with distutils. Fixing this bug might prove more difficult than adopting distutils and fixing it. Regardless, it's not something I'll be able to fix this afternoon as I'd hoped.
Here's the start of the idea I have in mind:

```diff
diff --git a/setuptools/dist.py b/setuptools/dist.py
index 7062ae8d..e92e7779 100644
--- a/setuptools/dist.py
+++ b/setuptools/dist.py
@@ -30,7 +30,7 @@ from . import SetuptoolsDeprecationWarning
 from setuptools.depends import Require
 from setuptools import windows_support
 from setuptools.monkey import get_unpatched
-from setuptools.config import parse_configuration
+from . import config
 import pkg_resources
 from .py36compat import Distribution_parse_config_files
 
@@ -563,8 +563,10 @@ class Distribution(Distribution_parse_config_files, _Distribution):
         """
         _Distribution.parse_config_files(self, filenames=filenames)
 
-        parse_configuration(self, self.command_options,
-                            ignore_option_errors=ignore_option_errors)
+        cfg = config.read_configuration(
+            filenames=filenames, ignore_option_errors=ignore_option_errors,
+        )
+        config.apply_configuration(self, cfg)
         self._finalize_requires()
 
     def parse_command_line(self):
```

I'm suggesting a refactor of `setuptools.config` that decouples reading the config from applying that config to a distribution.
This breaks `data_files`, since its target directories are keys `setup.cfg`, so the following:

```[options.data_files]
/some-path/to-dir =
    some/file
```

will install the file to `/some_path/to_dir/some/file`.
When this bug was filed, [setuptools relied on distutils](https://github.com/pypa/setuptools/blob/7843688bc33dd4e13e10130bc49da4c290fe7d7f/setuptools/dist.py#L564) to parse the config files. Since then, in 24be5abd4cbd9d84537c457456f841522d626e14, the code was inlined, so it should be straightforward to adjust the parsing behavior. I believe it's worth seriously considering just dropping that substitution.
As Jason mentioned, since the config parsing code is now within Setuptools, I believe this line is responsible for turning hyphens to underscores: https://github.com/pypa/setuptools/blob/main/setuptools/dist.py#L601. I wonder if we can simply remove the line, but I'm not sure if there are disadvantages to doing this.

Resolving patch:
diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
--- a/setuptools/command/easy_install.py
+++ b/setuptools/command/easy_install.py
@@ -1190,7 +1190,7 @@ def _set_fetcher_options(self, base):
         for key, val in ei_opts.items():
             if key not in fetch_directives:
                 continue
-            fetch_options[key.replace('_', '-')] = val[1]
+            fetch_options[key] = val[1]
         # create a settings dictionary suitable for `edit_config`
         settings = dict(easy_install=fetch_options)
         cfg_filename = os.path.join(base, 'setup.cfg')
diff --git a/setuptools/dist.py b/setuptools/dist.py
--- a/setuptools/dist.py
+++ b/setuptools/dist.py
@@ -598,7 +598,7 @@ def _parse_config_files(self, filenames=None):  # noqa: C901
                         continue
 
                     val = parser.get(section, opt)
-                    opt = opt.replace('-', '_')
+                    opt = self.dash_to_underscore_warning(opt, section)
                     opt_dict[opt] = (filename, val)
 
             # Make the ConfigParser forget everything (so we retain
@@ -623,6 +623,19 @@ def _parse_config_files(self, filenames=None):  # noqa: C901
             except ValueError as e:
                 raise DistutilsOptionError(e) from e
 
+    def dash_to_underscore_warning(self, opt, section):
+        if section in (
+            'options.extras_require', 'options.data_files',
+        ):
+            return opt
+        underscore_opt = opt.replace('-', '_')
+        if '-' in opt:
+            warnings.warn(
+                "Usage of dash-separated '%s' will not be supported in future "
+                "versions. Please use the underscore name '%s' instead"
+                % (opt, underscore_opt))
+        return underscore_opt
+
     # FIXME: 'Distribution._set_command_options' is too complex (14)
     def _set_command_options(self, command_obj, option_dict=None):  # noqa: C901
         """


Test patch:
diff --git a/setuptools/tests/test_build_ext.py b/setuptools/tests/test_build_ext.py
--- a/setuptools/tests/test_build_ext.py
+++ b/setuptools/tests/test_build_ext.py
@@ -104,7 +104,7 @@ def test_build_ext_config_handling(tmpdir_cwd):
         'setup.cfg': DALS(
             """
             [build]
-            build-base = foo_build
+            build_base = foo_build
             """),
     }
     path.build(files)
diff --git a/setuptools/tests/test_config.py b/setuptools/tests/test_config.py
--- a/setuptools/tests/test_config.py
+++ b/setuptools/tests/test_config.py
@@ -210,8 +210,8 @@ def test_aliases(self, tmpdir):
         fake_env(
             tmpdir,
             '[metadata]\n'
-            'author-email = test@test.com\n'
-            'home-page = http://test.test.com/test/\n'
+            'author_email = test@test.com\n'
+            'home_page = http://test.test.com/test/\n'
             'summary = Short summary\n'
             'platform = a, b\n'
             'classifier =\n'
@@ -507,6 +507,25 @@ def test_not_utf8(self, tmpdir):
             with get_dist(tmpdir):
                 pass
 
+    def test_dash_to_underscore_warning(self, tmpdir):
+        # dash_to_underscore_warning() is a method in setuptools.dist
+        # remove this test and method when dash convert to underscore in setup.cfg
+        # is no longer supported
+        fake_env(
+            tmpdir,
+            '[metadata]\n'
+            'author-email = test@test.com\n'
+            'maintainer_email = foo@foo.com\n'
+            )
+        msg = ("Usage of dash-separated 'author-email' will not be supported "
+               "in future versions. "
+               "Please use the underscore name 'author_email' instead")
+        with pytest.warns(UserWarning, match=msg):
+            with get_dist(tmpdir) as dist:
+                metadata = dist.metadata
+                assert metadata.author_email == 'test@test.com'
+                assert metadata.maintainer_email == 'foo@foo.com'
+
 
 class TestOptions:
 
@@ -772,6 +791,20 @@ def test_extras_require(self, tmpdir):
             }
             assert dist.metadata.provides_extras == set(['pdf', 'rest'])
 
+    def test_dash_preserved_extras_require(self, tmpdir):
+        fake_env(
+            tmpdir,
+            '[options.extras_require]\n'
+            'foo-a = foo\n'
+            'foo_b = test\n'
+        )
+
+        with get_dist(tmpdir) as dist:
+            assert dist.extras_require == {
+                'foo-a': ['foo'],
+                'foo_b': ['test']
+            }
+
     def test_entry_points(self, tmpdir):
         _, config = fake_env(
             tmpdir,