Repository: conan-io/conan

Issue:
[scm] Expressions not expanded when "auto" mode
Conan Version: 1.5.2

I just played around some more with the scm feature and found a (for me as user) surprising side effect. As soon as either the "url" or the "revision" field is in "auto" mode, not only the respective field is rewritten but also other fields in the scm structure are rewritten to their current value. For standard strings it does not matter, however, as soon as a function is called to provide the value, this becomes visible.

In more detail, exporting a recipe with the following scm structure yields the expected result and exactly the same structure is found in the conan cache.

    scm = {
        "type": "git",
        "subfolder": "repo",
        "username": tools.get_env("MY_MAGIC_USERNAME", "error"),
        "password": tools.get_env("MY_MAGIC_PASSWORD", "error"),
        "url": "http://example.com",
        "revision": "1234"
    }

However, as soon as I set, for example, the revision to "auto", all fields are evaluated and exported as their actual value at export time. This can then look something like the following:

    scm = {"password": "error",
           "revision": "1e25d838c88afaa6f190d12ea9dff9de36450f0c",
           "subfolder": "repo",
           "type": "git",
           "url": "http://example.com",
           "username": "error"}

What bothers me is that the behaviour feels a little bit inconsistent which makes me wonder if this is a bug or a feature?

On the one hand, I dislike that it happens as side effect of capturing the url/revision. On the other hand, I like the possibility to capture values at export time into the recipe. Having a generic measure like this would, for example, permit to easily implement #3150 inside the recipe without further conan support. I imagine something like an `expand` function, that can be used to initialize arbitrary properties, and is replaced with its result when the recipe is exported. Implementing #3150 would then look something like the following where `get_revision_from_branch` is defined as free function in the recipe.

    scm = {
        "type": "git",
        "url": "http://example.com",
        "revision": expand(get_revision_from_branch("feature/foo"))
    }

Similarly, #3160 could be extended to the following in order to remove the `get_version` call when consuming the recipe.

    version = expand(get_version())

What do you guys think about the idea? Is it worthwhile pursuing and can it be implemented at all?


Discussion:
Hi!

The problem is if you specify the following `scm` code in a `conanfile.py`:

```
scm = {
    "type": "git",
    "subfolder": "repo",
    "username": tools.get_env("MY_MAGIC_USERNAME", "error"),
    "password": tools.get_env("MY_MAGIC_PASSWORD", "error"),
    "url": "http://example.com",
    "revision": "1234"
}
```

the `tools.get_env` is evaluated when loading the class, there is nothing conan could do with it. Is python stuff.

The only way we could try to operate is if you declare a lambda function instead of a value, we could (theoretically speaking) evaluate the lambda only when accessing the field in the scm. So the only way I see to fix this is the opposite to your proposal, by default everything is expanded, but we could evaluate later a function.
I would also be fine with having expansion by default as long as it is consistent and there is a way to opt-out. If this is possible by specifying a lambda then this is fine by me.

The reason why I proposed an `expand` function as opt-in is the following. Without actually looking at conan's implementation for scm feature, my idea was to wrap the initialization value using something like an `ExpandValue` class via the `expand` function. Subsequently, the same logic, that is currently used to rewrite the scm block when url/revision is specified as "auto", could be used on every class member guarded by an `isInstance(member, ExpandValue)` check.
To summarize how the feature could look:

- The `scm` attribute in the conanfile could have some attributes with "expressions" or "lambdas" or something we don't have to evaluate at the `get_scm_data`. They should be treated as expressions (as they were strings => Note that a `lambda` as is won't work, because we don't want a `lambda object` we want the `get_revision_from_branch("feature/foo")` as a literal expression`
- So when the values are replaced, the data to be replaced is not the evaluation is the literal expression.
- To help understand it, an ugly example implementation could be:

```
scm = {
    "type": "git",
    "subfolder": "repo",
    "username": "#EXPRESSION#tools.get_env('MY_MAGIC_USERNAME', 'error')",
    "password":  "#EXPRESSION#tools.get_env('MY_MAGIC_PASSWORD', 'error')",
    "url": "http://example.com",
    "revision": "1234"
}
``` 

And in the replacement:
```
if attribute is string and starts with "#EXPRESSION#":
    attribute_str = value.split("#EXPRESSION#")[1]
```

Example implementation was done here: #3255 
So, @niosHD  in addition to "username" and "password" do you have any other use case for this "not expanded variables"? I can't imagine any other, maybe the feature could be simplified as declaring standard `CONAN_SCM_USERNAME` and `CONAN_SCM_PASSWORD` to be used when no username and password is specified?
Theoretically, the same problem exists for each field that exists within the `scm` property. For example, if someone wants to write a recipe where a command line variable permits to select if either an ssh or an https clone url is used, doing that is not straight forward possible using only the `scm` feature. Instead, I assume that a custom `source` method has to be used in addition to the `scm` auto deduction.

Still, to be perfectly honest and assuming that it works, I am completely fine with writing a custom `source` method in such scenarios. I just brought up the topic because the `scm` feature did not behave like I expected it to. Furthermore, I feel strongly that the whole `scm` auto deduction functionality is only a special case of a more generic and more useful feature (see #3338).

In summary, adding the eval like behavior with code in strings is in my opinion not very nice and most likely not worth the effort. It is probably best to document the whole situation as a limitation of the current `scm` feature and move on. Hopefully, a more generic way to capture and store information at export time makes the whole problem obsolete in the future.

Anyway, thank you @lasote for looking into this issue! I really appreciate the effort you guys put into conan and how you interact with the community.


Resolving patch:
diff --git a/conans/client/cmd/export.py b/conans/client/cmd/export.py
--- a/conans/client/cmd/export.py
+++ b/conans/client/cmd/export.py
@@ -69,12 +69,17 @@ def _capture_export_scm_data(conanfile, conanfile_dir, destination_folder, outpu
         os.unlink(scm_src_file)
 
     scm_data = get_scm_data(conanfile)
-    captured_revision = scm_data.capture_revision if scm_data else False
+    if not scm_data:
+        return None, False
 
-    if not scm_data or not (scm_data.capture_origin or scm_data.capture_revision):
-        return None, captured_revision
+    # Resolve SCMData in the user workspace (someone may want to access CVS or import some py)
+    captured_revision = scm_data.capture_revision
 
     scm = SCM(scm_data, conanfile_dir, output)
+    if scm_data.capture_origin or scm_data.capture_revision:
+        # Generate the scm_folder.txt file pointing to the src_path
+        src_path = scm.get_repo_root()
+        save(scm_src_file, src_path.replace("\\", "/"))
 
     if scm_data.url == "auto":
         origin = scm.get_qualified_remote_url()
@@ -84,17 +89,14 @@ def _capture_export_scm_data(conanfile, conanfile_dir, destination_folder, outpu
             output.warn("Repo origin looks like a local path: %s" % origin)
         output.success("Repo origin deduced by 'auto': %s" % origin)
         scm_data.url = origin
+
     if scm_data.revision == "auto":
         if not scm.is_pristine():
             output.warn("Repo status is not pristine: there might be modified files")
         scm_data.revision = scm.get_revision()
         output.success("Revision deduced by 'auto': %s" % scm_data.revision)
 
-    # Generate the scm_folder.txt file pointing to the src_path
-    src_path = scm.get_repo_root()
-    save(scm_src_file, src_path.replace("\\", "/"))
-    _replace_scm_data_in_conanfile(os.path.join(destination_folder, "conanfile.py"),
-                                   scm_data)
+    _replace_scm_data_in_conanfile(os.path.join(destination_folder, "conanfile.py"), scm_data)
 
     return scm_data, captured_revision
 
@@ -146,7 +148,6 @@ def _replace_scm_data_in_conanfile(conanfile_path, scm_data):
     content = content if not headers else ''.join(headers) + content
     save(conanfile_path, content)
 
-
 def _export_conanfile(conanfile_path, output, client_cache, conanfile, conan_ref, keep_source):
 
     exports_folder = client_cache.export(conan_ref)


Test patch:
diff --git a/conans/test/functional/old/scm_test.py b/conans/test/functional/old/scm_test.py
--- a/conans/test/functional/old/scm_test.py
+++ b/conans/test/functional/old/scm_test.py
@@ -473,6 +473,81 @@ def scm_serialization_test(self):
         data2 = json.loads(the_json)
         self.assertEquals(data, data2)
 
+    def test_git_delegated_function(self):
+        conanfile = """
+import os
+from conans import ConanFile
+from conans.client.tools.scm import Git
+
+def get_revision():
+    here = os.path.dirname(__file__)
+    git = Git(here)
+    return git.get_commit()
+
+def get_url():
+    def nested_url():
+        here = os.path.dirname(__file__)
+        git = Git(here)
+        return git.get_remote_url()
+    return nested_url()
+
+class MyLib(ConanFile):
+    name = "issue"
+    version = "3831"
+    scm = {'type': 'git', 'url': get_url(), 'revision': get_revision()}
+
+"""
+        self.client.save({"conanfile.py": conanfile})
+        path, commit = create_local_git_repo(folder=self.client.current_folder)
+        self.client.runner('git remote add origin "%s"' % path.replace("\\", "/"), cwd=path)
+
+        self.client.run("export . user/channel")
+        reference = ConanFileReference.loads("issue/3831@user/channel")
+        exported_conanfile = self.client.client_cache.conanfile(reference)
+        content = load(exported_conanfile)
+        self.assertIn(commit, content)
+
+    def test_delegated_python_code(self):
+        client = TestClient()
+        code_file = """
+import os
+from conans.tools import Git
+from conans import ConanFile
+
+def get_commit():
+    here = os.getcwd()
+    git = Git(here)
+    return git.get_commit()
+
+class MyLib(ConanFile):
+    pass
+"""
+        client.save({"conanfile.py": code_file})
+        client.run("export . tool/0.1@user/testing")
+
+        conanfile = """from conans import ConanFile, python_requires
+from conans.tools import load
+tool = python_requires("tool/0.1@user/testing")
+
+class MyLib(ConanFile):
+    scm = {'type': 'git', 'url': '%s', 'revision': tool.get_commit()}
+    def build(self):
+        self.output.info("File: {}".format(load("file.txt")))
+""" % client.current_folder.replace("\\", "/")
+
+        client.save({"conanfile.py": conanfile, "file.txt": "hello!"})
+        path, commit = create_local_git_repo(folder=client.current_folder)
+        client.runner('git remote add origin "%s"' % path.replace("\\", "/"), cwd=path)
+
+        client.run("export . pkg/0.1@user/channel")
+        reference = ConanFileReference.loads("pkg/0.1@user/channel")
+        exported_conanfile = client.client_cache.conanfile(reference)
+        content = load(exported_conanfile)
+        self.assertIn(commit, content)
+        # Check that everything works and is usable
+        client.run("install pkg/0.1@user/channel --build=missing")
+        self.assertIn("pkg/0.1@user/channel: File: hello!", client.out)
+
 
 @attr('svn')
 class SVNSCMTest(SVNLocalRepoTestCase):