Repository: conan-io/conan
Pull request: Add `auth_plugin` with tests
Changelog: Feature: New ``auth_remote.py`` plugin for custom user authentication to Conan remotes.
Changelog: Feature: New ``auth_source.py`` plugin for custom user authentication for generic downloads of sources.
Changelog: Bugfix: Allow `remote_login` accept patterns.
Docs: https://github.com/conan-io/docs/pull/3846

Close https://github.com/conan-io/conan/issues/4266
Close https://github.com/conan-io/conan/issues/4196


If we are going have two diferent plugins one for remote auth and another one for source credentials we might want to change this name, I have though `remote_auth_plugin` and `source_auth_plugin`

Problem statement:
Compatibility with keyring (libsecret) for user login credentials
It would be nice if I could store my credentials for a remote into a keychain such as `libsecret` like I can do with `git`.

[Question] How to use two factor authentication (2FA) like Google Authenticator with Conan?
Hi @memsharded and @lasote ,

I have a question concerning 2FA with conan.
Especially in enterprise environments a 2 factor authentication is often used to secure the access to various applications (like JIRA, Artifactory, etc.).
I didn't find any information about 2FA@conan.

Are there any experiences with Google Authenticator or similar applications (Microsoft Authenticator, FreeOTP, Keepass/KeyOTP, etc.) and how to use them with conan? 
If not, is there any feature planned?

Thanks in advance.
Best Aalmann

Hints from issue discussion:
I have changed the title, as keychain is more related to iOS apps, which does not relate here.

There is a python interface to libsecret thanks to Gnome: https://developer.gnome.org/libsecret/0.16/py-examples.html, which might be useful for this.

It sounds a bit very specific solution for one platform, I don't know if that will work in other Linuxes, for example, this should be checked.

Also, what would be the interface? Users would be able to opt-in to this? I don't think it should be enabled by default. How the interface would look like?


A section in Conan config `credential` with an entry `helper` which points to an executable that can receive commands to store and retrieve credentials. Absent by default which leads to no attempt by Conan to store/retrieve credentials.

It seems that if the command line interface is well defined for Conan credential helpers, it could be multiplatform and not even specific to keyring implementations of credential management.
> It seems that if the command line interface is well defined for Conan credential helpers, it could be multiplatform and not even specific to keyring implementations of credential management.

No, I don't mean the interface to the library :)

I meant the user interface. How the user should tell conan, in the command line, in ``conan.conf``, with an env-var... how to opt-in to store the credentials in the keyring. What would be the process to remove them from the keyring, if any (also, from the user point of view, not from the code)
Opt in is just specifying a credential helper.

I would say using the same interface as Git would be a good starting point: https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage
Ok, thanks for the feedback. That would be the ``conan.conf`` file then (which can be set with ``conan config`` too).

But I have checked it and bad news:
- There is no python package in PyPI for it: https://pypi.org/search/?q=libsecret. Not having it is almost a blocker, we cannot add dependencies to Conan that are not python packages, otherwise redistribution, creation of the installers, etc., becomes a nightmare.
- The python binding that comes with the linux system package in https://developer.gnome.org/libsecret/0.16/py-examples.html, seems to come licensed by a mixture of licenses, but including GPL terms. So conan (MIT) cannot use it.

Lets wait for feedback, to check if someone comes up with another solution that makes implementation reasonable. 

Without these, I think it would be a lot of work for the value, and my recommendation would be to not implement this feature at this moment.


Hi colleagues, 
Have you checked if `keyring` module fits your needs? (https://pypi.org/project/keyring/)
The use case that @Lawrencemm mentions can be covered up since it is able to use many different backends (depending on the OS but, from MacOS Keychain to Windows credential manager) or even adapt a customized one (https://pypi.org/project/keyring/#write-your-own-keyring-backend). BTW This module is licensed under MIT License terms, so may works. 

It is not that clear to me if he was really asking to provide also external connectivity to one developed by him. If so, i don't either see how to provide an interface for it.

Hi @alacasta 

Thanks for that pointer, very interesting, and it could facilitate things.
Still, I am seeing that in order to use it in mainstream systems as Ubuntu, the following is needed:

```bash
$ sudo apt install python3-venv libdbus-glib-1-dev
...
$ pip install secretstorage dbus-python
$ pip install keyring
```

So it would be very difficult to provide a fully automated pip install, but the users would still need to install those things from apt.

Maybe this would be a good candidate for a conan plugin (or maybe a hook? cc @jgsogo @danimtb)
Hi @memsharded, 

Not really sure if all this stuff is needed. As far as I remember from my experience using it on Ubuntu 16 was that worked out of the box. I think that all these recommendations were suggested for using it on Virutual Environments. Isn't it?
I think that is enough for this year so, happy new year Conan colleagues, you are doing an incredible job!!
In Windows it worked ok with the Windows Vault backend. In Linux:

```bash
>>> import keyring
>>> keyring.set_password("conan-test", "myuser", "mypassword")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/memsharded/venvs/conan27/local/lib/python2.7/site-packages/keyring/core.py", line 63, in set_password
    _keyring_backend.set_password(service_name, username, password)
  File "/home/memsharded/venvs/conan27/local/lib/python2.7/site-packages/keyring/backends/fail.py", line 23, in get_password
    raise RuntimeError(msg)
RuntimeError: No recommended backend was available. Install the keyrings.alt package if you want to use the non-recommended backends. See README.rst for details.
```

Also, I see there is no interface to get a user-password pair, only to query the password for a given user, but the user still needs to be defined. So not possible to automatically get the username, only a cached password.

With the above, if this will be considered, I now balance 99% towards doing this in a plugin/hook
And happy new year too @alacasta ! :) :) 
Hi @Aalmann!

Conan supports custom authentications: https://docs.conan.io/en/latest/uploading_packages/running_your_server.html#authentication

I've created a plugin for LDAP (as example): https://github.com/uilianries/conan-ldap-authentication

To implement a plugin for OAuth I think won't be complicated, based on my past work using [google-auth-oauthlib](https://github.com/GoogleCloudPlatform/google-auth-library-python-oauthlib). It uses [requests-oauthlib](https://github.com/requests/requests-oauthlib) to authenticate, so your plugin could check the token generated by the OAuth server.
 
According [Artifactory's docs](https://www.jfrog.com/confluence/display/RTF/OAuth+Integration) it has support for OAuth, but I don't know about 2FA.
I am not sure if I understand the question. Are you talking about the conan_server?
Because you comment you already have 2FA in Artifactory. 

If anything, it would be similar to Github: https://help.github.com/articles/accessing-github-using-two-factor-authentication/#using-two-factor-authentication-with-the-command-line

Artifactory already has user tokens, I think they might be used as password from the client side too, but need to check.
No not conan_server (and I think I didn't wrote it :smile:).

Our setup is: 
* Server side: Artifactory as binary server with user/password authentication (Atlassian Crowd directory integration) + "Timebased One Time Password" (e.g. Google Auth)
* Conan client installed via pip on user machine plus Google Authenticator as auth-token generator

So the question is: How to configure Conan client for using that setup.
I have no idea about it. But with Artifactory tokens, you can use the token like a password and Artifactory will accept it. So if with google auth code you can then get a token or something similar you might be able to use it. 

I think is not so simple, for Google 2FA you will need to create a Google Dev account and a new application there to provide a QR code to be used with Google Authenticator, and of course, Artifactory token will not work, bacause the token is based on Google side.

This feature should be developed by Artifatory team, where they could provide a QR Code on web interface to be added on Google Authenticator app. So the temporary token provided by the app should be used for Conan authentication. However, in my experience, performing 2FA authentication over CI jobs could cause headache for days.

But the question is, why do you need this? 2FA is really good for distributed applications that you can access around the world, like gmail, github, ... but Conan is a client and usually runs only in your company.
> Conan client installed via pip on user machine plus Google Authenticator as auth-token generator

Again, you will to create a plugin with all Google stuff. Fortunately, Google has a good python API and a big community to help. You will need to execute some steps in your plugin:

- The google authenticator will need a token distributed by server side, usually as QR Code, to be used as seed. So the first step is receive this token and store in somewhere. This could be solved at first time that you run `conan user`, the client should check if your authenticator is configured.
- When running `conan user` you will need to add your user and password, but as google authenticator generates temporary tokens, your plugin needs to check if your connection has expired (but you could set to keep connected as well) and reconnect if needed.
- You will need to check the provider's documentation to understand its API. I saw few providers supporting 2FA by REST, usually I've used Soup to parse the page and send the temporary token.
@lasote 
So the workflow would be: Login via Browser to handle user/password and Google Auth, grab the Artifactory token and use it as password? Isn't each session (conan and browser) different and so a problem for (re-)using the token at command line?

@uilianries 
Why we need this: VPN connections and everything is only available in corporate networks/infrastructure was "yesterday" (... sound weird 😃 ...). Today the most of the/our collaboration services (JIRA, Confluence, Bitbucket, Artifactory, etc.) are available via internet, to provide a better partner and team integration. But to better secure the access a second factor is required.
Currently this 2FA setup in combination with Artifactory is in planning/testing phase and I don't know the concrete implementation. But I want to be prepared and I want to be able to rise the hand if there is an issue with conan and the 2FA setup.
But what do you mean with create a "plugin"? I don't know any plugin mechanism in conan client (but in server). The only "extension points" I know are hooks and python requires and only hooks may be useful for that (pre_download, pre_upload).
> @lasote
So the workflow would be: Login via Browser to handle user/password and Google Auth, grab the Artifactory token and use it as password? Isn't each session (conan and browser) different and so a problem for (re-)using the token at command line?

I don't know if it makes sense. I would need an example of a client tool using Google Auth to login against a server. But eventually (irrespective of the authentication process with the server) you could get a token, so I thought that maybe it would be Artifactory responsibility to generate an access token for Conan. Once the token is created it will be valid until it expires, but from my perspective, it has nothing to do anymore with Google Auth. But again, I have no idea about this.
I almost forgot about VPN case, now your case is more clear. When I said about "plugin" actually it means docs.conan.io/en/latest/uploading_packages/running_your_server.html#authentication It works like an extension, as you commented.


@uilianries 
But this is only available for conan_server and we don't use it. We use Artifactory. 

@lasote 
Sadly I can't provide you such an example.

Commit message: Add `auth_plugin` with tests

Patch to review:
diff --git a/conan/internal/cache/home_paths.py b/conan/internal/cache/home_paths.py
index ba2402ded2a..562dfa994ff 100644
--- a/conan/internal/cache/home_paths.py
+++ b/conan/internal/cache/home_paths.py
@@ -55,6 +55,10 @@ def profiles_path(self):
     def profile_plugin_path(self):
         return os.path.join(self._home, _EXTENSIONS_FOLDER, _PLUGINS, "profile.py")
 
+    @property
+    def auth_plugin_path(self):
+        return os.path.join(self._home, _EXTENSIONS_FOLDER, _PLUGINS, "auth.py")
+
     @property
     def sign_plugin_path(self):
         return os.path.join(self._home, _EXTENSIONS_FOLDER, _PLUGINS, "sign", "sign.py")
diff --git a/conans/client/migrations.py b/conans/client/migrations.py
index ddde216ac69..26ec23cfea6 100644
--- a/conans/client/migrations.py
+++ b/conans/client/migrations.py
@@ -52,6 +52,9 @@ def _apply_migrations(self, old_version):
         # Update profile plugin
         from conan.internal.api.profile.profile_loader import migrate_profile_plugin
         migrate_profile_plugin(self.cache_folder)
+        # Update auth plugin
+        from conans.client.rest.remote_credentials import migrate_auth_plugin
+        migrate_auth_plugin(self.cache_folder)
 
         if old_version and old_version < "2.0.14-":
             _migrate_pkg_db_lru(self.cache_folder, old_version)
diff --git a/conans/client/rest/remote_credentials.py b/conans/client/rest/remote_credentials.py
index 33dbe638d08..cf6d6652a0d 100644
--- a/conans/client/rest/remote_credentials.py
+++ b/conans/client/rest/remote_credentials.py
@@ -4,16 +4,29 @@
 
 from jinja2 import Template
 
+from conan.internal.cache.home_paths import HomePaths
+
+from conans.client.loader import load_python_file
 from conan.api.output import ConanOutput
 from conans.client.userio import UserInput
-from conans.errors import ConanException
+from conans.errors import ConanException, scoped_traceback
 from conans.util.files import load
 
+_default_auth_plugin = """
+# This file was generated by Conan. Remove this comment if you edit this file or Conan
+# will destroy your changes.
+
+def auth_plugin(remote, user=None, password=None):
+    # TODO: Connect this function to your own keyring for user login credentials
+    return None, None
+"""
+
 
 class RemoteCredentials:
     def __init__(self, cache_folder, global_conf):
         self._global_conf = global_conf
         self._urls = {}
+        self.auth_plugin_path = HomePaths(cache_folder).auth_plugin_path
         creds_path = os.path.join(cache_folder, "credentials.json")
         if not os.path.exists(creds_path):
             return
@@ -32,7 +45,19 @@ def auth(self, remote, user=None, password=None):
         if user is not None and password is not None:
             return user, password
 
-        # First prioritize the cache "credentials.json" file
+        # First get the auth_plugin
+        auth_plugin = self._load_auth_plugin()
+        if auth_plugin is not None:
+            try:
+                plugin_user, plugin_password = auth_plugin(remote, user=user, password=password)
+            except Exception as e:
+                msg = f"Error while processing 'auth.py' plugin"
+                msg = scoped_traceback(msg, e, scope="/extensions/plugins")
+                raise ConanException(msg)
+            if plugin_user and plugin_password:
+                return plugin_user, plugin_password
+
+        # Then prioritize the cache "credentials.json" file
         creds = self._urls.get(remote)
         if creds is not None:
             try:
@@ -66,3 +91,18 @@ def _get_env(remote, user):
         if passwd:
             ConanOutput().info("Got password '******' from environment")
         return user, passwd
+
+    def _load_auth_plugin(self):
+        if not os.path.exists(self.auth_plugin_path):
+            raise ConanException("The 'auth.py' plugin file doesn't exist. If you want "
+                                 "to disable it, edit its contents instead of removing it")
+
+        mod, _ = load_python_file(self.auth_plugin_path)
+        if hasattr(mod, "auth_plugin"):
+            return mod.auth_plugin
+
+def migrate_auth_plugin(cache_folder):
+    from conans.client.migrations import update_file
+
+    auth_plugin_file = HomePaths(cache_folder).auth_plugin_path
+    update_file(auth_plugin_file, _default_auth_plugin)
diff --git a/test/integration/configuration/test_auth_plugin.py b/test/integration/configuration/test_auth_plugin.py
new file mode 100644
index 00000000000..8472a9a86a9
--- /dev/null
+++ b/test/integration/configuration/test_auth_plugin.py
@@ -0,0 +1,60 @@
+import os
+import textwrap
+
+import pytest
+
+from conan.test.utils.tools import TestClient
+from conans.util.files import save
+
+
+class TestErrorsAuthPlugin:
+    """ when the plugin fails, we want a clear message and a helpful trace
+    """
+    def test_error_profile_plugin(self):
+        c = TestClient(default_server_user=True)
+        auth_plugin = textwrap.dedent("""\
+            def auth_plugin(remote, user=None, password=None):
+                raise Exception("Test Error")
+            """)
+        save(os.path.join(c.cache.plugins_path, "auth.py"), auth_plugin)
+        c.run("remote logout default")
+        c.run("remote login default", assert_error=True)
+        assert "Error while processing 'auth.py' plugin" in c.out
+        assert "Test Error" in c.out
+
+    def test_remove_plugin_file(self):
+        c = TestClient()
+        c.run("version")  # to trigger the creation
+        os.remove(os.path.join(c.cache.plugins_path, "auth.py"))
+        c.run("remote add default http://fake")
+        c.run("remote login default", assert_error=True)
+        assert "ERROR: The 'auth.py' plugin file doesn't exist" in c.out
+
+    @pytest.mark.parametrize("password", ["password", "bad-password"])
+    def test_profile_plugin_direct_credentials(self, password):
+        should_fail = password == "bad-password"
+        c = TestClient(default_server_user=True)
+        auth_plugin = textwrap.dedent(f"""\
+            def auth_plugin(remote, user=None, password=None):
+                return "admin", "{password}"
+            """)
+        save(os.path.join(c.cache.plugins_path, "auth.py"), auth_plugin)
+        c.run("remote logout default")
+        c.run("remote login default", assert_error=should_fail)
+        if should_fail:
+            assert "ERROR: Wrong user or password. [Remote: default]" in c.out
+        else:
+            assert "Changed user of remote 'default' from 'None' (anonymous) to 'admin' (authenticated)" in c.out
+
+    def test_profile_plugin_fallback(self):
+        c = TestClient(default_server_user=True)
+        auth_plugin = textwrap.dedent("""\
+                def auth_plugin(remote, user=None, password=None):
+                    return None, None
+                """)
+        save(os.path.join(c.cache.plugins_path, "auth.py"), auth_plugin)
+        c.run("remote logout default")
+        c.run("remote login default")
+        # As the auth plugin is not returning any password the code is falling back to the rest of
+        # the input methods in this case the stdin provided by TestClient.
+        assert "Changed user of remote 'default' from 'None' (anonymous) to 'admin' (authenticated)" in c.out


Code review comments:
diff_hunk: @@ -52,6 +52,9 @@ def _apply_migrations(self, old_version):
         # Update profile plugin
         from conan.internal.api.profile.profile_loader import migrate_profile_plugin
         migrate_profile_plugin(self.cache_folder)
+        # Update auth plugin
+        from conans.client.rest.remote_credentials import migrate_auth_plugin
+        migrate_auth_plugin(self.cache_folder)
path: conans/client/migrations.py
text: @user1:
Conan will not provide a default plugin, so a migration will not be necessary

diff_hunk: @@ -66,3 +91,18 @@ def _get_env(remote, user):
         if passwd:
             ConanOutput().info("Got password '******' from environment")
         return user, passwd
+
+    def _load_auth_plugin(self):
path: conans/client/rest/remote_credentials.py
text: @user1:
Let's have this as a free function, and pass the auth_plugin_path as an argument, instead of saving it to the class

diff_hunk: @@ -4,16 +4,29 @@
 
 from jinja2 import Template
 
+from conan.internal.cache.home_paths import HomePaths
+
+from conans.client.loader import load_python_file
 from conan.api.output import ConanOutput
 from conans.client.userio import UserInput
-from conans.errors import ConanException
+from conans.errors import ConanException, scoped_traceback
 from conans.util.files import load
 
+_default_auth_plugin = """
path: conans/client/rest/remote_credentials.py
text: @user1:
We don't need an empty default plugin, better consider the non-existence of the file as user didn't provide one and do nothing.

diff_hunk: @@ -4,16 +4,29 @@
 
 from jinja2 import Template
 
+from conan.internal.cache.home_paths import HomePaths
+
+from conans.client.loader import load_python_file
 from conan.api.output import ConanOutput
 from conans.client.userio import UserInput
-from conans.errors import ConanException
+from conans.errors import ConanException, scoped_traceback
 from conans.util.files import load
 
+_default_auth_plugin = """
+# This file was generated by Conan. Remove this comment if you edit this file or Conan
+# will destroy your changes.
+
+def auth_plugin(remote, user=None, password=None):
path: conans/client/rest/remote_credentials.py
text: @user1:
That fact that this is only the `remote.name` might hamper users from coming up with useful solutions, and they have no (easy) way to instantiate the proper `conan_api` either to re-resolve it. Maybe we can check if passing the whole `Remote` object would be better?

@user2:
Yes, using the remote ``url`` for example might be useful for users.