Repository: conan-io/conan

Issue:
[Commands] Validate remote URL
Conan version: 1.7.3

Testing conan behind a proxy let me to an error when I specified a wrong URL for my conan remote. I simply forgot to specify the protocol in the remote configuration:

`conan remote update localhost 127.0.0.1:9300`
Should have been:
`conan remote update localhost http://127.0.0.1:9300`

The wrong remote specification together with the no_proxy information:
`echo $no_proxy`
`localhost,127.0.0.1,localaddress`
 or in conan.conf below `[proxies]`
`no_proxy_match = localhost,127.0.0.1,localaddress`

I got the error:
`conan upload <test_package> -r=localhost`
`ERROR: inet_aton() argument 1 must be string, not None`

It took me a while to track down the issue to `requests.utils.should_bypass_proxies(...)` where the url `127.0.0.1:9300/v1/ping` is parsed. Of course, if the request package handles this kind of url better, there is still an error from conan:
`ERROR: No connection adapters were found for 127.0.0.1:9300/v1/ping`
The error still is not clear enough IMHO. 

I think it would be more user friendly if the remote url is validated when added or the error message should be more clear what's wrong. (In addition to the package request, which should not fail that way ;-))


Discussion:
Hi Florian, 

Thanks for the detailed report. We need to have a look at this, because it seems a good idea, but I am also afraid about pure validation checks. There is always a chance that they throw false positives, blocking someone to add their remote because it didn't pass some validation rule, even if it is valid for they case (custom DNS, local modification to hosts addresses, etc). For those cases I usually recommend a warning, not a blocking error.

Regarding the error later in the upload, yes, we will check it, the message might be improved.

Hi James,

I fully agree with you, that false positives are possible and appreciate the idea of a warning, so that user can simply ignore it, if they know it will work in their environment.
Ok, lets do a warning to ``conan remote`` commands, that would be easy and might help some users that did typos on the URL. 

Resolving patch:
diff --git a/conans/client/cache/remote_registry.py b/conans/client/cache/remote_registry.py
--- a/conans/client/cache/remote_registry.py
+++ b/conans/client/cache/remote_registry.py
@@ -1,6 +1,7 @@
 import json
 import os
 from collections import OrderedDict, namedtuple
+from six.moves.urllib.parse import urlparse
 
 from conans.errors import ConanException, NoRemoteAvailable
 from conans.util.config_parser import get_bool_from_text_value
@@ -253,6 +254,18 @@ def __init__(self, cache, output):
         self._output = output
         self._filename = cache.registry_path
 
+    def _validate_url(self, url):
+        """ Check if URL contains protocol and address
+
+        :param url: URL to be validated
+        """
+        if url:
+            address = urlparse(url)
+            if not all([address.scheme, address.netloc]):
+                self._output.warn("The URL '%s' is invalid. It must contain scheme and hostname." % url)
+        else:
+            self._output.warn("The URL is empty. It must contain scheme and hostname.")
+
     def load_remotes(self):
         if not os.path.exists(self._filename):
             self._output.warn("Remotes registry file missing, "
@@ -265,6 +278,7 @@ def load_remotes(self):
         return remotes
 
     def add(self, remote_name, url, verify_ssl=True, insert=None, force=None):
+        self._validate_url(url)
         remotes = self.load_remotes()
         renamed = remotes.add(remote_name, url, verify_ssl, insert, force)
         remotes.save(self._filename)
@@ -278,6 +292,7 @@ def add(self, remote_name, url, verify_ssl=True, insert=None, force=None):
                             pkg_metadata.remote = remote_name
 
     def update(self, remote_name, url, verify_ssl=True, insert=None):
+        self._validate_url(url)
         remotes = self.load_remotes()
         remotes.update(remote_name, url, verify_ssl, insert)
         remotes.save(self._filename)


Test patch:
diff --git a/conans/test/functional/command/remote_test.py b/conans/test/functional/command/remote_test.py
--- a/conans/test/functional/command/remote_test.py
+++ b/conans/test/functional/command/remote_test.py
@@ -379,3 +379,16 @@ def package_refs_test(self):
     def missing_subarguments_test(self):
         self.client.run("remote", assert_error=True)
         self.assertIn("ERROR: Exiting with code: 2", self.client.out)
+
+    def test_invalid_url(self):
+        self.client.run("remote add foobar foobar.com")
+        self.assertIn("WARN: The URL 'foobar.com' is invalid. It must contain scheme and hostname.",
+                      self.client.user_io.out)
+        self.client.run("remote list")
+        self.assertIn("foobar.com", self.client.out)
+
+        self.client.run("remote update foobar pepe.org")
+        self.assertIn("WARN: The URL 'pepe.org' is invalid. It must contain scheme and hostname.",
+                      self.client.user_io.out)
+        self.client.run("remote list")
+        self.assertIn("pepe.org", self.client.out)
diff --git a/conans/test/functional/configuration/registry_test.py b/conans/test/functional/configuration/registry_test.py
--- a/conans/test/functional/configuration/registry_test.py
+++ b/conans/test/functional/configuration/registry_test.py
@@ -127,3 +127,23 @@ def insert_test(self):
                          Remote("repo2", "url2", True),
                          Remote("conan.io", "https://server.conan.io", True),
                          Remote("repo3", "url3", True)])
+
+    def test_remote_none(self):
+        """ RemoteRegistry should be able to deal when the URL is None
+        """
+        f = os.path.join(temp_folder(), "add_none_test")
+        Remotes().save(f)
+        cache = ClientCache(os.path.dirname(f), TestBufferConanOutput())
+        registry = cache.registry
+
+        registry.add("foobar", None)
+        self.assertEqual(list(registry.load_remotes().values()),
+                         [("conan-center", "https://conan.bintray.com", True),
+                          ("foobar", None, True)])
+        self.assertIn("WARN: The URL is empty. It must contain scheme and hostname.", cache._output)
+        registry.remove("foobar")
+
+        registry.update("conan-center", None)
+        self.assertEqual(list(registry.load_remotes().values()),
+                         [("conan-center", None, True)])
+        self.assertIn("WARN: The URL is empty. It must contain scheme and hostname.", cache._output)
\ No newline at end of file