Repository: cekit/cekit

Issue:
add authentication to artifacts
**Describe the solution you'd like**
I'm missing the possibility to add authentication (like tokens, username/password) when using artifacts in cekit. The option to use URL artifacts is pretty nice, if the repository is accessable without any authentication.

This would be a nice addition:

```diff
   name: "jolokia"
   url: "https://github.com/rhuss/jolokia/releases/download/v1.3.6/jolokia-1.3.6-bin.tar.gz"
   # The md5 checksum of the artifact
   md5: "75e5b5ba0b804cd9def9f20a70af649f"
+ token: verysecure
```
or for older repos
```diff
   name: "jolokia"
   url: "https://github.com/rhuss/jolokia/releases/download/v1.3.6/jolokia-1.3.6-bin.tar.gz"
   # The md5 checksum of the artifact
   md5: "75e5b5ba0b804cd9def9f20a70af649f"
+ username: downloader
+ password: secureone
```

**Additional context**
Maybe I've missed the part in the documentation or an already open issue. Any help and suggestions are appreciated.


Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/cekit/cekit/pull/918?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cekit) Report
Attention: Patch coverage is `92.30769%` with `1 line` in your changes missing coverage. Please review.
> Project coverage is 90.29%. Comparing base [(`5efda67`)](https://app.codecov.io/gh/cekit/cekit/commit/5efda67ad5f22c6647397cc2e7ce49bd337488bd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cekit) to head [(`b4425e7`)](https://app.codecov.io/gh/cekit/cekit/commit/b4425e7b3ce2cc2f7fb0d50d990e2dfffd2e78b2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cekit).

| [Files](https://app.codecov.io/gh/cekit/cekit/pull/918?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cekit) | Patch % | Lines |
|---|---|---|
| [cekit/builders/osbs.py](https://app.codecov.io/gh/cekit/cekit/pull/918?src=pr&el=tree&filepath=cekit%2Fbuilders%2Fosbs.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cekit#diff-Y2VraXQvYnVpbGRlcnMvb3Nicy5weQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/cekit/cekit/pull/918?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cekit) |

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           develop     #918      +/-   ##
===========================================
+ Coverage    90.26%   90.29%   +0.02%     
===========================================
  Files           42       42              
  Lines         3256     3266      +10     
===========================================
+ Hits          2939     2949      +10     
  Misses         317      317              
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/cekit/cekit/pull/918?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cekit).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cekit).


Patch:
diff --git a/cekit/builders/osbs.py b/cekit/builders/osbs.py
index d9250c0..2f52eb8 100644
--- a/cekit/builders/osbs.py
+++ b/cekit/builders/osbs.py
@@ -62,7 +62,7 @@ class OSBSBuilder(Builder):
         deps = {}
 
         if CONFIG.get("common", "redhat"):
-            if CONFIG.get("common", "stage"):
+            if params.get("stage"):
                 fedpkg = "rhpkg-stage"
                 koji = "brewkoji-stage"
                 koji_executable = "brew-stage"
diff --git a/cekit/tools.py b/cekit/tools.py
index 997b0b2..2f38214 100644
--- a/cekit/tools.py
+++ b/cekit/tools.py
@@ -1,3 +1,4 @@
+import base64
 import importlib
 import logging
 import os
@@ -7,7 +8,7 @@ import subprocess
 import sys
 from typing import Any, Mapping, Sequence
 from urllib.parse import urlparse
-from urllib.request import urlopen
+from urllib.request import Request, urlopen
 
 import click
 import yaml
@@ -56,7 +57,24 @@ def download_file(url: str, destination: str) -> None:
             ctx.check_hostname = False
             ctx.verify_mode = ssl.CERT_NONE
 
-        res = urlopen(url, context=ctx)
+        request: Request = Request(url)
+        if config.get("common", "url_authentication"):
+            domains = dict(
+                [
+                    x.split("#")
+                    for x in config.get("common", "url_authentication").split(";")
+                ]
+            )
+            for d, a in domains.items():
+                if parsed_url.hostname == d:
+                    logger.debug(
+                        f"Located matching hostname '{d}' to add authentication with username '{a.split(':')[0]}'."
+                    )
+                    b64auth = base64.b64encode(a.encode("latin-1")).decode()
+                    request.add_header("Authorization", f"Basic {b64auth}")
+                    break
+
+        res = urlopen(request, context=ctx)
 
         if res.getcode() != 200:
             raise CekitError(f"Could not download file from {url}")
diff --git a/docs/descriptor/includes/artifacts.rst b/docs/descriptor/includes/artifacts.rst
index 20eb218..5ee7499 100644
--- a/docs/descriptor/includes/artifacts.rst
+++ b/docs/descriptor/includes/artifacts.rst
@@ -226,6 +226,10 @@ This approach relies on :doc:`/handbook/caching` to provide the artifact.
 The ``source-url`` and source checksum (which may be ``source-md5``, ``source-sha1`` or
 ``source-sha256`` are optional and denote where the sources may be found.
 
+.. note::
+
+   See :doc:`/handbook/configuration` for information on how URL authentication may be configured.
+
 Path artifacts
 ******************
 
diff --git a/docs/handbook/configuration.rst b/docs/handbook/configuration.rst
index bc77d06..a953f21 100644
--- a/docs/handbook/configuration.rst
+++ b/docs/handbook/configuration.rst
@@ -150,3 +150,23 @@ Example
         [common]
         fetch_artifact_domains = https://www.foo.bar/my-path,https://www.example.com
 
+URL Artifact Authentication
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Key
+    ``url_authentication``
+
+Description
+    This option makes it possible to provide a username/password on a per-domain basis when downloading URL based artifacts. Multiple domains each with their own username/password may be specified with a ``;`` deliminator while the domain is separated from the ``username:password`` combination via a ``#`` deliminator.
+
+Default
+    Not set
+
+Example
+    .. code-block:: ini
+
+        [common]
+        url_authentication=example.com#username:password;www.foo.org#username:password
+
+.. note::
+    Due to the separators being used (``;``, ``#``, ``:``) these characters cannot be used within a password.


Test patch:
diff --git a/tests/test_unit_resource.py b/tests/test_unit_resource.py
index 6155046..05fa05c 100644
--- a/tests/test_unit_resource.py
+++ b/tests/test_unit_resource.py
@@ -1,5 +1,6 @@
 import logging
 import os
+from urllib.request import Request
 
 import pytest
 import yaml
@@ -122,14 +123,16 @@ def test_fetching_with_ssl_verify(mocker):
     get_mock_ssl(mocker, ctx)
     mock_urlopen = get_mock_urlopen(mocker)
 
-    res = create_resource({"name": "file", "url": "https:///dummy"})
+    res = create_resource({"name": "file", "url": "https://dummy"})
 
     try:
         res.copy()
     except Exception:
         pass
 
-    mock_urlopen.assert_called_with("https:///dummy", context=ctx)
+    request: Request = mock_urlopen.call_args[0][0]
+    mock_urlopen.assert_called_with(request, context=ctx)
+    assert request.get_full_url() == "https://dummy"
     assert ctx.check_hostname is True
     assert ctx.verify_mode == 1
 
@@ -141,21 +144,23 @@ def test_fetching_disable_ssl_verify(mocker):
     ctx = get_ctx(mocker)
     get_mock_ssl(mocker, ctx)
 
-    res = create_resource({"name": "file", "url": "https:///dummy"})
+    res = create_resource({"name": "file", "url": "https://dummy"})
 
     try:
         res.copy()
     except Exception:
         pass
 
-    mock_urlopen.assert_called_with("https:///dummy", context=ctx)
+    request: Request = mock_urlopen.call_args[0][0]
+    mock_urlopen.assert_called_with(request, context=ctx)
+    assert request.get_full_url() == "https://dummy"
 
     assert ctx.check_hostname is False
     assert ctx.verify_mode == 0
 
 
 def test_fetching_bad_status_code():
-    res = create_resource({"name": "file", "url": "http:///dummy"})
+    res = create_resource({"name": "file", "url": "http://dummy"})
     with pytest.raises(CekitError):
         res.copy()
 
@@ -172,7 +177,7 @@ def test_fetching_file_exists_but_used_as_is(mocker):
     res = create_resource(
         {
             "name": "file",
-            "url": "http:///dummy",
+            "url": "http://dummy",
             "md5": "d41d8cd98f00b204e9800998ecf8427e",
         }
     )
@@ -191,13 +196,15 @@ def test_fetching_file_exists_fetched_again(mocker):
 
     with open("file", "w") as f:  # noqa: F841
         pass
-    res = create_resource({"name": "file", "url": "http:///dummy", "md5": "123456"})
+    res = create_resource({"name": "file", "url": "http://dummy", "md5": "123456"})
     with pytest.raises(CekitError):
         # Checksum will fail, because the "downloaded" file
         # will not have md5 equal to 123456. We need investigate
         # mocking of requests get calls to do it properly
         res.copy()
-    mock_urlopen.assert_called_with("http:///dummy", context=ctx)
+    request: Request = mock_urlopen.call_args[0][0]
+    mock_urlopen.assert_called_with(request, context=ctx)
+    assert request.get_full_url() == "http://dummy"
 
 
 def test_fetching_file_exists_no_hash_fetched_again(mocker):
@@ -212,13 +219,15 @@ def test_fetching_file_exists_no_hash_fetched_again(mocker):
     with open("file", "w") as f:  # noqa: F841
         pass
 
-    res = create_resource({"name": "file", "url": "http:///dummy"})
+    res = create_resource({"name": "file", "url": "http://dummy"})
 
     with pytest.raises(CekitError):
         # url is not valid so we get error, but we are not interested
         # in it. We just need to check that we attempted to downlad.
         res.copy()
-    mock_urlopen.assert_called_with("http:///dummy", context=ctx)
+    request: Request = mock_urlopen.call_args[0][0]
+    mock_urlopen.assert_called_with(request, context=ctx)
+    assert request.get_full_url() == "http://dummy"
 
 
 def test_generated_url_without_cacher():
@@ -304,7 +313,9 @@ def test_url_resource_download_cleanup_after_failure(mocker, tmpdir, caplog):
     )
     assert f"Removing incompletely downloaded '{targetfile}' file" in caplog.text
 
-    urlopen_class_mock.assert_called_with("http://server.org/dummy", context=mocker.ANY)
+    request: Request = urlopen_class_mock.call_args[0][0]
+    urlopen_class_mock.assert_called_with(request, context=mocker.ANY)
+    assert request.get_full_url() == "http://server.org/dummy"
     os_remove_mock.assert_called_with(targetfile)
 
 
@@ -332,7 +343,9 @@ def test_copy_plain_resource_with_cacher(mocker, tmpdir):
     res.copy(str(tmpdir))
 
     substitute_cache_url_mock.assert_called_once_with(None)
-    urlopen_class_mock.assert_called_with("http://cache/abc", context=ctx)
+    request: Request = urlopen_class_mock.call_args[0][0]
+    urlopen_class_mock.assert_called_with(request, context=ctx)
+    assert request.get_full_url() == "http://cache/abc"
 
 
 def test_copy_plain_resource_from_brew(mocker, tmpdir):
@@ -362,7 +375,9 @@ def test_copy_plain_resource_from_brew(mocker, tmpdir):
 
     mock_get_brew_url.assert_called_once_with("5b9164ad6f496d9dee12ec7634ce253f")
     assert res._Resource__substitute_cache_url.call_count == 0
-    urlopen_class_mock.assert_called_with("http://cache/abc", context=ctx)
+    request: Request = urlopen_class_mock.call_args[0][0]
+    urlopen_class_mock.assert_called_with(request, context=ctx)
+    assert request.get_full_url() == "http://cache/abc"
 
 
 def test_override_resource_remove_chksum():
@@ -401,3 +416,27 @@ def test_override_resource_remove_chksum():
     assert "sha1" not in overrides["artifacts"][0]
     assert "sha256" not in overrides["artifacts"][0]
     assert "sha512" not in overrides["artifacts"][0]
+
+
+def test_fetching_file_with_authentication(mocker):
+    """
+    It should download the file again, because available
+    file locally doesn't match checksum.
+    """
+    mock_urlopen = get_mock_urlopen(mocker)
+    config.cfg["common"]["url_authentication"] = "dummy.com#username:password"
+    ctx = get_ctx(mocker)
+    get_mock_ssl(mocker, ctx)
+
+    with open("file", "w") as f:  # noqa: F841
+        pass
+    res = create_resource({"name": "file", "url": "http://dummy.com", "md5": "123456"})
+    with pytest.raises(CekitError):
+        # Checksum will fail, because the "downloaded" file
+        # will not have md5 equal to 123456. We need investigate
+        # mocking of requests get calls to do it properly
+        res.copy()
+    request: Request = mock_urlopen.call_args[0][0]
+    mock_urlopen.assert_called_with(request, context=ctx)
+    assert request.get_full_url() == "http://dummy.com"
+    assert request.get_header("Authorization") == "Basic dXNlcm5hbWU6cGFzc3dvcmQ="
diff --git a/tests/test_validate.py b/tests/test_validate.py
index ea34081..ee24f1b 100644
--- a/tests/test_validate.py
+++ b/tests/test_validate.py
@@ -5,6 +5,7 @@ import re
 import shutil
 import sys
 import uuid
+from urllib.request import Request
 
 import pytest
 import yaml
@@ -688,9 +689,8 @@ def test_run_load_remote_override(tmpdir, mocker):
     )
 
     assert check_dockerfile(image_dir, "USER 4321", "Containerfile")
-    mock_urlopen.assert_called_with(
-        "https://example.com/overrides.yaml", context=mocker.ANY
-    )
+    request: Request = mock_urlopen.call_args[0][0]
+    assert request.get_full_url() == "https://example.com/overrides.yaml"
 
 
 def test_run_load_remote_file_override(tmpdir, mocker):