Repository: conan-io/conan

Issue:
scm "url" = "auto" vs GitLab CI runner
My CI scripts simultaneously build Windows and Linux binaries on an internal GitLab server using Gitlab runners. Both Windows and Linux runner uses docker if that matters. And most the time one of them fails to upload a package to an internal Artifactory CE repo:

```
Remote manifest:
Time: 2018-11-26 11:45:30
conanfile.py, MD5: 3dd4f4769e515e72d79466d548dd2d32

Local manifest:
Time: 2018-11-26 11:44:54
conanfile.py, MD5: d7535264b9001b772fdf0317e1f95aef

Local 'conanfile.py' using '\n' line-ends
Remote 'conanfile.py' using '\n' line-ends

----------------------------------------
Traceback (most recent call last):
  File "build.py", line 44, in <module>
    main()
  File "build.py", line 41, in main
    builder.run()
  File "/usr/local/lib/python2.7/dist-packages/cpt/packager.py", line 443, in run
    self.run_builds(base_profile_name=base_profile_name)
  File "/usr/local/lib/python2.7/dist-packages/cpt/packager.py", line 518, in run_builds
    r.run()
  File "/usr/local/lib/python2.7/dist-packages/cpt/runner.py", line 88, in run
    self._upload, package_id)
  File "/usr/local/lib/python2.7/dist-packages/cpt/uploader.py", line 43, in upload_packages
    retry=int(self._upload_retry))
  File "/usr/local/lib/python2.7/dist-packages/conans/client/conan_api.py", line 88, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/conans/client/conan_api.py", line 818, in upload
    retry_wait, integrity_check, policy, remote_name, query=query)
  File "/usr/local/lib/python2.7/dist-packages/conans/client/cmd/uploader.py", line 68, in upload
    integrity_check, policy, remote_name, recorder)
  File "/usr/local/lib/python2.7/dist-packages/conans/client/cmd/uploader.py", line 90, in _upload
    remote_manifest = self._check_recipe_date(conan_ref, recipe_remote)
  File "/usr/local/lib/python2.7/dist-packages/conans/client/cmd/uploader.py", line 165, in _check_recipe_date
    (remote_recipe_manifest.time, local_manifest.time))
conans.errors.ConanException: Remote recipe is newer than local recipe: 
 Remote date: 1543232730
 Local date: 1543232694
```
If I rerun the failing job manually, it succeeds. Any Idea how to fix/work around this? Both Windows and Linux docker instances are running on the same physical server, so time sync issues are hardly the problem. Even if they are, can I somehow prevent conan from detecting that difference and failing with an exception?



Discussion:
My initial analysis was totally wrong.  Here is an updated one. I have a recipe with
```python
class BuildConan(ConanFile):
    scm = {
        "type": "git",
        "url": "auto",
        "revision": "auto",
        "submodule": "recursive"
    }
```
that I build with GitLab CI. `"url": "auto"` is supposed to be replaced by the real git repo url. But when GitLab runner executes CI job git url contain a unique job token needed to access a git repo: `http://gitlab-ci-token:t26Shcf6ip7mwYzszMCL@git.host.local/user/repo.git`. This token changes each time CI job is run. So concurrent jobs generate different `conanfile.py` and I see the above output.

Replacing `auto` with the real url `http://git.host.local/user/repo.git` (gitlab-ci-token stripped) seems to help. Maybe conan should do that itself? (or at least issue a warning if it sees `something@` in the `scm.url`?)

Thanks for the explanation. I think we cannot assume that Conan should clear always the credentials, it could happen a user actually wants them. Maybe we could add a new argument to the `scm`, something like `clear_url_credentials: True/False` (Default True?).

I think that those url credentials should be removed always... if the user wants to save inside the recipe the `username` and `password`, there are [fields in the `scm` dict dedicated to that info](https://docs.conan.io/en/latest/reference/conanfile/attributes.html#scm-attribute).

Yes, a new argument looks good to me.  But maybe also check for some well known CI credentials like `gitlab-ci-token:xxxxxxxx@` and strip them unconditionally (maybe with a warning)?
Sorry, missed @jgsogo comment. For me, "removing always" (with a warning that url credentials have been removed) is even better.
@db4 Do you happen to have a link to the Dockerfile you used to create the Docker image for Windows? I have been trying to test it but it looks way more complicated than the Linux docker file, getting and installing msvc compiler is just a pain...
@AliAskari You can find my Docker file here: https://github.com/conan-io/conan-docker-tools/issues/50#issuecomment-419881449
@jgsogo ok, we can remove them always. We can add the flag later if it is requested.
@AliAskari about docker windows, there is a WIP [here](https://github.com/conan-io/conan-docker-tools/pull/61). Please try to avoid mixing topics in the same issue if they are not related.
Hi! I think #3183 feature from @jgsogo could help with this issue.  You could write a tinny function now to remove the credentials from the URL. I think that the mechanism is generic and good enough. Could this issue be closed when merged? 

e.g:

```
import os
from conans import ConanFile
from conans.client.tools.scm import Git

    
def get_url():
    here = os.path.dirname(__file__)
    git = Git(here)
    tmp = git.get_remote_url()
    # REPLACE HERE YOUR URL
    return tmp
    
class MyLib(ConanFile):
    name = "issue"
    version = "3831"
    scm = {'type': 'git', 'url': get_url(), 'revision': "auto"}
```



@lasote I would still prefer automatic credential removal by conan, proposed by @jgsogo. Yes, your function is tiny, but one should not forget to copy/paste it each time a new conan package is created. But if it's too hard, of course explicit `git.get_remote_url()` is a way to go.
Thanks for the feedback
Hey @lasote ,

I have exactly the same problem with gitlab and it already hurt alot across my projects because CI get really noisy.

I also think that a directl url cleanup in conan would be great, because copy paste the `get_ur`l it into every recipe is not that pratical. But I think you can make that opt-in, so f.e.

```py
class BuildConan(ConanFile):
    scm = {
        "type": "git",
        "url": "auto",
        "revision": "auto",
        "submodule": "recursive",
        "credential_removal" : True # <= new option
    }
```

Maybe you can call it `remove_sensitve_data` or something like that.

The nice thing is that it is an opt-in feature, so every recipe author can decide if removal is required or not. It also wont fail in older conan version,but not removal would happen.

If you don't like the option, it would be nice to have a "ready to use" function in `tools`.

Resolving patch:
diff --git a/conans/client/cmd/export.py b/conans/client/cmd/export.py
--- a/conans/client/cmd/export.py
+++ b/conans/client/cmd/export.py
@@ -82,7 +82,7 @@ def _capture_export_scm_data(conanfile, conanfile_dir, destination_folder, outpu
         save(scm_src_file, src_path.replace("\\", "/"))
 
     if scm_data.url == "auto":
-        origin = scm.get_qualified_remote_url()
+        origin = scm.get_qualified_remote_url(remove_credentials=True)
         if not origin:
             raise ConanException("Repo origin cannot be deduced by 'auto'")
         if scm.is_local_repository():
diff --git a/conans/client/tools/scm.py b/conans/client/tools/scm.py
--- a/conans/client/tools/scm.py
+++ b/conans/client/tools/scm.py
@@ -50,6 +50,15 @@ def get_url_with_credentials(self, url):
         url = url.replace("://", "://" + user_enc + ":" + pwd_enc + "@", 1)
         return url
 
+    @classmethod
+    def _remove_credentials_url(cls, url):
+        parsed = urlparse(url)
+        netloc = parsed.hostname
+        if parsed.port:
+            netloc += ":{}".format(parsed.port)
+        replaced = parsed._replace(netloc=netloc)
+        return replaced.geturl()
+
 
 class Git(SCMBase):
     cmd_command = "git"
@@ -117,7 +126,7 @@ def excluded_files(self):
             ret = []
         return ret
 
-    def get_remote_url(self, remote_name=None):
+    def get_remote_url(self, remote_name=None, remove_credentials=False):
         self._check_git_repo()
         remote_name = remote_name or "origin"
         remotes = self.run("remote -v")
@@ -125,6 +134,8 @@ def get_remote_url(self, remote_name=None):
             name, url = remote.split(None, 1)
             if name == remote_name:
                 url, _ = url.rsplit(None, 1)
+                if remove_credentials and not os.path.exists(url):  # only if not local
+                    url = self._remove_credentials_url(url)
                 return url
         return None
 
@@ -261,12 +272,15 @@ def excluded_files(self):
                 excluded_list.append(os.path.normpath(filepath))
         return excluded_list
 
-    def get_remote_url(self):
-        return self._show_item('url')
+    def get_remote_url(self, remove_credentials=False):
+        url = self._show_item('url')
+        if remove_credentials and not os.path.exists(url):  # only if not local
+            url = self._remove_credentials_url(url)
+        return url
 
-    def get_qualified_remote_url(self):
+    def get_qualified_remote_url(self, remove_credentials=False):
         # Return url with peg revision
-        url = self.get_remote_url()
+        url = self.get_remote_url(remove_credentials=remove_credentials)
         revision = self.get_last_changed_revision()
         return "{url}@{revision}".format(url=url, revision=revision)
         
diff --git a/conans/model/scm.py b/conans/model/scm.py
--- a/conans/model/scm.py
+++ b/conans/model/scm.py
@@ -79,8 +79,8 @@ def checkout(self):
             output += self.repo.checkout(url=self._data.url, revision=self._data.revision)
         return output
 
-    def get_remote_url(self):
-        return self.repo.get_remote_url()
+    def get_remote_url(self, remove_credentials):
+        return self.repo.get_remote_url(remove_credentials=remove_credentials)
 
     def get_revision(self):
         return self.repo.get_revision()
@@ -91,11 +91,11 @@ def is_pristine(self):
     def get_repo_root(self):
         return self.repo.get_repo_root()
 
-    def get_qualified_remote_url(self):
+    def get_qualified_remote_url(self, remove_credentials):
         if self._data.type == "git":
-            return self.repo.get_remote_url()
+            return self.repo.get_remote_url(remove_credentials=remove_credentials)
         else:
-            return self.repo.get_qualified_remote_url()
+            return self.repo.get_qualified_remote_url(remove_credentials=remove_credentials)
 
     def is_local_repository(self):
         return self.repo.is_local_repository()


Test patch:
diff --git a/conans/test/functional/scm/__init__.py b/conans/test/functional/scm/__init__.py
new file mode 100644
diff --git a/conans/test/functional/scm/test_url_auto.py b/conans/test/functional/scm/test_url_auto.py
new file mode 100644
--- /dev/null
+++ b/conans/test/functional/scm/test_url_auto.py
@@ -0,0 +1,35 @@
+# coding=utf-8
+
+import unittest
+import textwrap
+
+from conans.test.utils.tools import TestClient
+from conans.test.utils.tools import NO_SETTINGS_PACKAGE_ID, TestClient, TestServer, \
+    create_local_git_repo
+from conans.model.ref import ConanFileReference, PackageReference
+
+
+class RemoveCredentials(unittest.TestCase):
+
+    conanfile = textwrap.dedent("""\
+        from conans import ConanFile
+        
+        class Lib(ConanFile):
+            scm = {"type": "git", "url": "auto"}
+            
+    """)
+
+    def setUp(self):
+        self.ref = ConanFileReference.loads("lib/1.0@lasote/testing")
+        self.path, _ = create_local_git_repo({"conanfile.py": self.conanfile})
+        self.client = TestClient()
+        self.client.current_folder = self.path
+        self.client.runner("git remote add origin https://url.to.be.sustituted", cwd=self.path)
+
+    def test_https(self):
+        expected_url = 'https://myrepo.com.git'
+        origin_url = 'https://username:password@myrepo.com.git'
+
+        self.client.runner("git remote set-url origin {}".format(origin_url), cwd=self.path)
+        self.client.run("export . {}".format(self.ref))
+        self.assertIn("Repo origin deduced by 'auto': {}".format(expected_url), self.client.out)
diff --git a/conans/test/unittests/client/tools/scm/__init__.py b/conans/test/unittests/client/tools/scm/__init__.py
new file mode 100644
diff --git a/conans/test/unittests/client/tools/scm/test_git.py b/conans/test/unittests/client/tools/scm/test_git.py
new file mode 100644
--- /dev/null
+++ b/conans/test/unittests/client/tools/scm/test_git.py
@@ -0,0 +1,21 @@
+# coding=utf-8
+
+import unittest
+
+from conans.client.tools.scm import Git
+from conans.test.utils.tools import temp_folder
+
+
+class GitRemoteUrlTest(unittest.TestCase):
+
+    def test_remove_credentials(self):
+        """ Check that the 'remove_credentials' argument is taken into account """
+        expected_url = 'https://myrepo.com/path/to/repo.git'
+        origin_url = 'https://username:password@myrepo.com/path/to/repo.git'
+
+        git = Git(folder=temp_folder())
+        git.run("init .")
+        git.run("remote add origin {}".format(origin_url))
+
+        self.assertEqual(git.get_remote_url(), origin_url)
+        self.assertEqual(git.get_remote_url(remove_credentials=True), expected_url)
diff --git a/conans/test/unittests/client/tools/scm/test_scm_base.py b/conans/test/unittests/client/tools/scm/test_scm_base.py
new file mode 100644
--- /dev/null
+++ b/conans/test/unittests/client/tools/scm/test_scm_base.py
@@ -0,0 +1,47 @@
+# coding=utf-8
+
+import unittest
+
+from conans.client.tools.scm import SCMBase
+
+
+class RemoveCredentialsTest(unittest.TestCase):
+
+    def test_http(self):
+        expected_url = 'https://myrepo.com/path/to/repo.git'
+        test_urls = ['https://myrepo.com/path/to/repo.git',
+                     'https://username:password@myrepo.com/path/to/repo.git',
+                     'https://username@myrepo.com/path/to/repo.git',
+                     'https://gitlab-ci-token:1324@myrepo.com/path/to/repo.git',
+                     ]
+
+        for it in test_urls:
+            self.assertEqual(expected_url, SCMBase._remove_credentials_url(it))
+
+    def test_http_with_port_number(self):
+        self.assertEqual('https://myrepo.com:8000/path/to/repo.git',
+                         SCMBase._remove_credentials_url(
+                             'https://username@myrepo.com:8000/path/to/repo.git'))
+
+    def test_ssh(self):
+        # Here, for ssh, we don't want to remove the user ('git' in this example)
+        self.assertEqual('git@github.com:conan-io/conan.git',
+                         SCMBase._remove_credentials_url(
+                             'git@github.com:conan-io/conan.git'))
+
+    def test_local_unix(self):
+        self.assertEqual('file:///srv/git/project.git',
+                         SCMBase._remove_credentials_url('file:///srv/git/project.git'))
+        self.assertEqual('file:///srv/git/PROJECT.git',
+                         SCMBase._remove_credentials_url('file:///srv/git/PROJECT.git'))
+
+    def test_local_windows(self):
+        self.assertEqual('file:///c:/srv/git/PROJECT',
+                         SCMBase._remove_credentials_url('file:///c:/srv/git/PROJECT'))
+        self.assertEqual('file:///C:/srv/git/PROJECT',
+                         SCMBase._remove_credentials_url('file:///C:/srv/git/PROJECT'))
+
+    def test_svn_ssh(self):
+        self.assertEqual('svn+ssh://10.106.191.164/home/svn/shproject',
+                         SCMBase._remove_credentials_url(
+                             'svn+ssh://username:password@10.106.191.164/home/svn/shproject'))
\ No newline at end of file
diff --git a/conans/test/unittests/client/tools/scm/test_svn.py b/conans/test/unittests/client/tools/scm/test_svn.py
new file mode 100644
--- /dev/null
+++ b/conans/test/unittests/client/tools/scm/test_svn.py
@@ -0,0 +1,23 @@
+# coding=utf-8
+
+import unittest
+
+from mock import patch
+
+from conans.client.tools.scm import SVN
+from conans.test.utils.tools import temp_folder
+
+
+class SVNRemoteUrlTest(unittest.TestCase):
+
+    def test_remove_credentials(self):
+        """ Check that the 'remove_credentials' argument is taken into account """
+        expected_url = 'https://myrepo.com/path/to/repo'
+        origin_url = 'https://username:password@myrepo.com/path/to/repo'
+
+        svn = SVN(folder=temp_folder())
+
+        # Mocking, as we cannot change SVN remote to a non-existing url
+        with patch.object(svn, '_show_item', return_value=origin_url):
+            self.assertEqual(svn.get_remote_url(), origin_url)
+            self.assertEqual(svn.get_remote_url(remove_credentials=True), expected_url)