Repository: ansible/ansible-builder

Issue:
ansible-builder failed to build EE for cisco.aci because of Double requirement given
### Issue
ansible-builder returned the following error when trying to build EE for AAP 2.1 with cisco.aci collection

~~~
+ pip3 install --cache-dir=/output/wheels -r /tmp/src/requirements.txt
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
ERROR: Double requirement given: python_dateutil (from -r /tmp/src/requirements.txt (line 19)) (already in python-dateutil>=2.7.0 (from -r /tmp/src/requirements.txt (line 5)), name='python-dateutil')
[3/3] STEP 1/5: FROM registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8:latest
Error: error building at STEP "RUN assemble": error while running runtime: exit status 1
~~~

### Step to reproduce

1) Using following version of podman and ansible-builder
~~~
# podman --version
podman version 3.4.2

# ansible-builder --version
1.0.1
~~~

2) Prepare the following files(execution-environment.yml/requirements.yml) before the build:

~~~
# cat execution-environment.yml
---
version: 1

build_arg_defaults:
  EE_BASE_IMAGE: 'registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8:latest'
dependencies:
  galaxy: requirements.yml

# cat requirements.yml
---
collections:
  - name: cisco.aci
    version: 2.1.0
~~~

3) build the EE with ansible-builder command
~~~
  # ansible-builder build -v 3
~~~

### Additional Info
- This issue only occurs when specifying cisco.aci version 2.1.0 collection for requirements.yml.
- There is no issue when specifying version 2.0.0
- 2.1.0 has following requirements.txt but 2.0.0 does not
~~~
lxml
cryptography
pyOpenSSL
python_dateutil
xmljson
~~~
- When using ee-minimal-rhel8, the build has been completed without issue
- We found there is old pip version in `EE_BUILDER_IMAGE` provided from registry.redhat.io. There is no issue when specifying `EE_BUILDER_IMAGE: 'quay.io/ansible/ansible-builder:latest'`. So I think we need to upgrade pip in builder image.

~~~
# podman run -it registry.redhat.io/ansible-automation-platform-21/ansible-builder-rhel8:latest pip3 --version
pip 20.2.4 from /usr/lib/python3.8/site-packages/pip (python 3.8)

# podman run -it quay.io/ansible/ansible-builder:latest pip3 --version
pip 22.0.3 from /usr/local/lib/python3.8/site-packages/pip (python 3.8)
~~~

Discussion:
softwarefactory-project-zuul[bot]: Build succeeded.

- [ansible-build-python-tarball ](https://ansible.softwarefactory-project.io/zuul/build/3875f5a11f6a4a278e4da1600e29e430) : SUCCESS in 3m 57s

softwarefactory-project-zuul[bot]: Build succeeded.

- [ansible-build-python-tarball ](https://ansible.softwarefactory-project.io/zuul/build/546d817ce4ff4869a64a5577d99bd0ba) : SUCCESS in 4m 39s

Shrews: @shanemcd When you return from PTO, can you review this for us?

Patch:
diff --git a/ansible_builder/requirements.py b/ansible_builder/requirements.py
index 2c85317..5b99a76 100644
--- a/ansible_builder/requirements.py
+++ b/ansible_builder/requirements.py
@@ -1,5 +1,6 @@
 import logging
 import requirements
+from pkg_resources import safe_name
 
 
 logger = logging.getLogger(__name__)
@@ -37,6 +38,8 @@ def sanitize_requirements(collection_py_reqs):
     for collection, lines in collection_py_reqs.items():
         try:
             for req in requirements.parse('\n'.join(lines)):
+                if req.specifier:
+                    req.name = safe_name(req.name)
                 req.collections = [collection]  # add backref for later
                 if req.name is None:
                     consolidated.append(req)


Test patch:
diff --git a/test/data/ansible_collections/other/reqfile/requirements.txt b/test/data/ansible_collections/other/reqfile/requirements.txt
new file mode 100644
index 0000000..3f3d8ac
--- /dev/null
+++ b/test/data/ansible_collections/other/reqfile/requirements.txt
@@ -0,0 +1,1 @@
+python_dateutil    # intentional underscore
diff --git a/test/data/ansible_collections/test/reqfile/requirements.txt b/test/data/ansible_collections/test/reqfile/requirements.txt
index 694334b..81590b6 100644
--- a/test/data/ansible_collections/test/reqfile/requirements.txt
+++ b/test/data/ansible_collections/test/reqfile/requirements.txt
@@ -1,2 +1,3 @@
 pytz
+python-dateutil>=2.8.2    # intentional dash
 -r extra_req.txt
diff --git a/test/integration/test_introspect_cli.py b/test/integration/test_introspect_cli.py
index 7852acd..f21f73b 100644
--- a/test/integration/test_introspect_cli.py
+++ b/test/integration/test_introspect_cli.py
@@ -35,6 +35,7 @@ def test_introspect_write_python(cli, data_dir, tmp_path):
     assert dest_file.read_text() == '\n'.join([
         'pyvcloud>=14  # from collection test.metadata',
         'pytz  # from collection test.reqfile',
+        'python-dateutil>=2.8.2  # from collection test.reqfile',
         'tacacs_plus  # from collection test.reqfile',
         'pyvcloud>=18.0.10  # from collection test.reqfile',
         '',
@@ -48,7 +49,8 @@ def test_introspect_write_python_and_sanitize(cli, data_dir, tmp_path):
     assert dest_file.read_text() == '\n'.join([
         'pyvcloud>=14,>=18.0.10  # from collection test.metadata,test.reqfile',
         'pytz  # from collection test.reqfile',
-        'tacacs_plus  # from collection test.reqfile',
+        'python-dateutil>=2.8.2  # from collection test.reqfile',
+        'tacacs-plus  # from collection test.reqfile',
         '',
     ])
 
diff --git a/test/unit/test_introspect.py b/test/unit/test_introspect.py
index 027d141..d950a04 100644
--- a/test/unit/test_introspect.py
+++ b/test/unit/test_introspect.py
@@ -13,7 +13,10 @@ def test_multiple_collection_metadata(data_dir):
     assert files == {'python': [
         'pyvcloud>=14,>=18.0.10  # from collection test.metadata,test.reqfile',
         'pytz  # from collection test.reqfile',
-        'tacacs_plus  # from collection test.reqfile'
+        # python-dateutil should appear only once even though referenced in
+        # multiple places, once with a dash and another with an underscore in the name.
+        'python-dateutil>=2.8.2  # from collection test.reqfile',
+        'tacacs-plus  # from collection test.reqfile'
     ], 'system': [
         'subversion [platform:rpm]  # from collection test.bindep',
         'subversion [platform:dpkg]  # from collection test.bindep'