Repository: conan-io/conan

Issue:
[feature] Supporting mirrors in tools.get
It would be handy if `tools.get()` could support mirrors to have fallback locations to try to download a file from if the first one fails.

### Problem

My main use case for such a feature is currently linked to conan-center-index: the way to retrieve the sources today is to put link to the compressed sources in `conandata.yml`, and the `source()` method of the corresponding recipe generally consumes the data from the file as follows:

```cpp
tools.get(**self.conan_data["sources"][self.version])
```

However today I need to build `sqlite3` with a specific configuration, but www.sqlite.org is down, which means that Conan fails to retrieve the sources and my whole build is kind of blocked. If `tools.get` supported mirrors it would be possible to add a list of mirrors to the `conandata.yml` to have other locations to retrieve the sources from when the first one fails.

### Solution

While it would be possible to have `tools.get()` accept a list of URLs in `filename`, I believe that it might get users confused and think that it can retrieve a list of files from the given URLs, so it would probably more sensible to add a `mirrors` parameter accepting a list of URLs. Since YAML accept lists it means that a full list of mirrors could be forwarded directly from the `conandata.yml` file too.

If `tools.get()` gets a `mirrors=` parameter, then it makes sense to mirror the feature and add it to `tools.download()` too.

- [x] I've read the [CONTRIBUTING guide](https://github.com/conan-io/conan/blob/develop/.github/CONTRIBUTING.md).



Discussion:
I am not sure what to think about this feature. I understand the idea, but I am not sure of the real value in practice:

- How often is www.sqlite.org down, or other equivalent source origins? From our experience packaging things in Conan-center, this is extremely unusual. Builds fail for other reasons, but not this
- How often, when a site is down, there are reliable and trusted mirrors? What are the download URLs for sqlite.org from the mirrors?

If I am building from sources sqlite packages for my company, the first thing I would most likely do is to do a copy of those sources to my own private location (a generic repo in Artifactory, typically), so I don't have a problem. This was the original intention of ``conandata.yml``, to have a place where you can easily change this source origin without even needing to edit the recipe.

So, while the feature per-se is fine, and I wouldn't oppose to this functionality, seems a bit on the edge, I am lacking some evidence about the magnitude of the pain to solve.
All good points. Concerning the official mirrors I was thinking about the number of projects that host their source control and downloads on their own servers but have official mirrors on GitHub or GitLab, I don't know whether there might be other use cases for CCI (I think that we already use such mirrors there rather than their upstream repositories, and if GitHub itself is down I've got to admit that chances are that some other dependency might not build).
Yes, exactly, if Github is down, anything else will fail anyway, and the number of cases where a solid mirror is found are very limited. I think we can wait until we have more evidence of failing builds that a mirror solution would be consistently better.

Resolving patch:
diff --git a/conans/client/tools/net.py b/conans/client/tools/net.py
--- a/conans/client/tools/net.py
+++ b/conans/client/tools/net.py
@@ -12,10 +12,13 @@ def get(url, md5='', sha1='', sha256='', destination=".", filename="", keep_perm
         overwrite=False, auth=None, headers=None):
     """ high level downloader + unzipper + (optional hash checker) + delete temporary zip
     """
-    if not filename and ("?" in url or "=" in url):
-        raise ConanException("Cannot deduce file name from url. Use 'filename' parameter.")
 
-    filename = filename or os.path.basename(url)
+    url = [url] if not isinstance(url, (list, tuple)) else url
+    if not filename and ("?" in url[0] or "=" in url[0]):
+        raise ConanException("Cannot deduce file name from the url: '{}'. Use 'filename' "
+                             "parameter.".format(url[0]))
+    filename = filename or os.path.basename(url[0])
+
     download(url, filename, out=output, requester=requester, verify=verify, retry=retry,
              retry_wait=retry_wait, overwrite=overwrite, auth=auth, headers=headers,
              md5=md5, sha1=sha1, sha256=sha256)
@@ -49,7 +52,30 @@ def ftp_download(ip, filename, login='', password=''):
 
 def download(url, filename, verify=True, out=None, retry=None, retry_wait=None, overwrite=False,
              auth=None, headers=None, requester=None, md5='', sha1='', sha256=''):
+    """Retrieves a file from a given URL into a file with a given filename.
+       It uses certificates from a list of known verifiers for https downloads,
+       but this can be optionally disabled.
 
+    :param url: URL to download. It can be a list, which only the first one will be downloaded, and
+                the follow URLs will be used as mirror in case of download error.
+    :param filename: Name of the file to be created in the local storage
+    :param verify: When False, disables https certificate validation
+    :param out: An object with a write() method can be passed to get the output. stdout will use if
+                not specified
+    :param retry: Number of retries in case of failure. Default is overriden by general.retry in the
+                  conan.conf file or an env variable CONAN_RETRY
+    :param retry_wait: Seconds to wait between download attempts. Default is overriden by
+                       general.retry_wait in the conan.conf file or an env variable CONAN_RETRY_WAIT
+    :param overwrite: When True, Conan will overwrite the destination file if exists. Otherwise it
+                      will raise an exception
+    :param auth: A tuple of user and password to use HTTPBasic authentication
+    :param headers: A dictionary with additional headers
+    :param requester: HTTP requests instance
+    :param md5: MD5 hash code to check the downloaded file
+    :param sha1: SHA-1 hash code to check the downloaded file
+    :param sha256: SHA-256 hash code to check the downloaded file
+    :return: None
+    """
     out = default_output(out, 'conans.client.tools.net.download')
     requester = default_requester(requester, 'conans.client.tools.net.download')
     from conans.tools import _global_config as config
@@ -60,21 +86,38 @@ def download(url, filename, verify=True, out=None, retry=None, retry_wait=None,
     retry_wait = retry_wait if retry_wait is not None else config.retry_wait
     retry_wait = retry_wait if retry_wait is not None else 5
 
-    downloader = FileDownloader(requester=requester, output=out, verify=verify, config=config)
+    url = [url] if not isinstance(url, (list, tuple)) else url
     checksum = sha256 or sha1 or md5
-    # The download cache is only used if a checksum is provided, otherwise, a normal download
+
+    downloader = FileDownloader(requester=requester, output=out, verify=verify, config=config)
     if config and config.download_cache and checksum:
         downloader = CachedFileDownloader(config.download_cache, downloader, user_download=True)
-        downloader.download(url, filename, retry=retry, retry_wait=retry_wait, overwrite=overwrite,
-                            auth=auth, headers=headers, md5=md5, sha1=sha1, sha256=sha256)
-    else:
-        downloader.download(url, filename, retry=retry, retry_wait=retry_wait, overwrite=overwrite,
-                            auth=auth, headers=headers)
-        if md5:
-            check_md5(filename, md5)
-        if sha1:
-            check_sha1(filename, sha1)
-        if sha256:
-            check_sha256(filename, sha256)
 
-    out.writeln("")
+    def _download_file(downloader, url):
+        # The download cache is only used if a checksum is provided, otherwise, a normal download
+        if isinstance(downloader, CachedFileDownloader):
+            downloader.download(url, filename, retry=retry, retry_wait=retry_wait,
+                                overwrite=overwrite, auth=auth, headers=headers, md5=md5,
+                                sha1=sha1, sha256=sha256)
+        else:
+            downloader.download(url, filename, retry=retry, retry_wait=retry_wait,
+                                overwrite=overwrite, auth=auth, headers=headers)
+            if md5:
+                check_md5(filename, md5)
+            if sha1:
+                check_sha1(filename, sha1)
+            if sha256:
+                check_sha256(filename, sha256)
+        out.writeln("")
+
+    for index, url_it in enumerate(url):
+        try:
+            _download_file(downloader, url_it)
+            break
+        except ConanException as error:
+            message = "Could not download from the URL {}: {}.".format(url_it, str(error))
+            if len(url) == 1:
+                raise ConanException(message)
+            out.warn(message + " Trying another mirror.")
+            if (index + 1) == len(url):
+                raise ConanException("All downloads from ({}) URLs have failed.".format(len(url)))


Test patch:
diff --git a/conans/test/functional/cache/download_cache_test.py b/conans/test/functional/cache/download_cache_test.py
--- a/conans/test/functional/cache/download_cache_test.py
+++ b/conans/test/functional/cache/download_cache_test.py
@@ -122,7 +122,8 @@ def source(self):
            """ % http_server.port)
         client.save({"conanfile.py": conanfile})
         client.run("source .", assert_error=True)
-        self.assertIn("ConanException: md5 signature failed", client.out)
+        self.assertIn("ConanException: Could not download from the URL http://localhost:{}/myfile."
+                      "txt: md5 signature failed for".format(http_server.port), client.out)
         self.assertIn("Provided signature: kk", client.out)
         self.assertIn("Computed signature: 9893532233caff98cd083a116b013c0b", client.out)
         self.assertEqual(1, len(os.listdir(cache_folder)))  # Nothing was cached
diff --git a/conans/test/unittests/util/tools_test.py b/conans/test/unittests/util/tools_test.py
--- a/conans/test/unittests/util/tools_test.py
+++ b/conans/test/unittests/util/tools_test.py
@@ -13,6 +13,7 @@
 from mock.mock import mock_open, patch
 from nose.plugins.attrib import attr
 from parameterized import parameterized
+from requests.models import Response
 
 from conans.client import tools
 from conans.client.cache.cache import CONAN_CONF
@@ -472,7 +473,9 @@ def download_retries_errors_test(self):
         self.assertEqual(str(out).count("Waiting 1 seconds to retry..."), 2)
 
         # Not found error
-        with six.assertRaisesRegex(self, NotFoundException, "Not found: "):
+        with six.assertRaisesRegex(self, ConanException,
+                                   "Could not download from the URL http://google.es/FILE_NOT_FOUND:"
+                                   " Not found: http://google.es/FILE_NOT_FOUND."):
             tools.download("http://google.es/FILE_NOT_FOUND",
                            os.path.join(temp_folder(), "README.txt"), out=out,
                            requester=requests,
@@ -661,9 +664,10 @@ def error_url():
 
         out = TestBufferConanOutput()
         # Test: File name cannot be deduced from '?file=1'
-        with six.assertRaisesRegex(self, ConanException,
-                                   "Cannot deduce file name from url. Use 'filename' parameter."):
+        with self.assertRaises(ConanException) as error:
             tools.get("http://localhost:%s/?file=1" % thread.port, output=out)
+        self.assertIn("Cannot deduce file name from the url: 'http://localhost:{}/?file=1'."
+                      " Use 'filename' parameter.".format(thread.port), str(error.exception))
 
         # Test: Works with filename parameter instead of '?file=1'
         with tools.chdir(tools.mkdir_tmp()):
@@ -729,6 +733,56 @@ def get_file():
 
         thread.stop()
 
+    @patch("conans.client.tools.net.unzip")
+    def test_get_mirror(self, _):
+        """ tools.get must supports a list of URLs. However, only one must be downloaded.
+        """
+
+        class MockRequester(object):
+            def __init__(self):
+                self.count = 0
+                self.fail_first = False
+                self.fail_all = False
+
+            def get(self, *args, **kwargs):
+                self.count += 1
+                resp = Response()
+                resp._content = b'{"results": []}'
+                resp.headers = {"Content-Type": "application/json"}
+                resp.status_code = 200
+                if (self.fail_first and self.count == 1) or self.fail_all:
+                    resp.status_code = 408
+                return resp
+
+        file = "test.txt.gz"
+        out = TestBufferConanOutput()
+        urls = ["http://localhost:{}/{}".format(8000 + i, file) for i in range(3)]
+
+        # Only the first file must be downloaded
+        with tools.chdir(tools.mkdir_tmp()):
+            requester = MockRequester()
+            tools.get(urls, requester=requester, output=out, retry=0, retry_wait=0)
+            self.assertEqual(1, requester.count)
+
+        # Fail the first, download only the second
+        with tools.chdir(tools.mkdir_tmp()):
+            requester = MockRequester()
+            requester.fail_first = True
+            tools.get(urls, requester=requester, output=out, retry=0, retry_wait=0)
+            self.assertEqual(2, requester.count)
+            self.assertIn("WARN: Could not download from the URL {}: Error 408 downloading file {}."
+                          " Trying another mirror."
+                          .format(urls[0], urls[0]), out)
+
+        # Fail all downloads
+        with tools.chdir(tools.mkdir_tmp()):
+            requester = MockRequester()
+            requester.fail_all = True
+            with self.assertRaises(ConanException) as error:
+                tools.get(urls, requester=requester, output=out, retry=0, retry_wait=0)
+            self.assertEqual(3, requester.count)
+            self.assertIn("All downloads from (3) URLs have failed.", str(error.exception))
+
     def check_output_runner_test(self):
         import tempfile
         original_temp = tempfile.gettempdir()