Repository: conan-io/conan

Issue:
SCM auto mode improvements
This is a new issue to manage together #5126 and #4732 (probably some other).

**Current pains**

- `scm_folder.txt` is tricky and ugly (internal concern mostly).
- Since the usage of the local sources should be avoided when running a `conan install` command (see next section), could be still a good mechanism to take benefit of the optimization without taking the sources from the local folder but not cloning the repo. (@niosHD at https://github.com/conan-io/conan/issues/4732)
- A recipe containing `scm` with a real revision exported is dangerous if the folder was not pristine because the recipe could be uploaded and the CI could run something different from the expected. (@Mark-Hatch-Bose at https://github.com/conan-io/conan/issues/5126)

**Already solved pains to be kept**

- The local sources shouldn't be used in a `conan install --build XX` because it could take sources too much changed or belonging even to a different version. It should use the sources only for `conan create`. Solved at #4218

**Proposition**

- The `scm` auto will copy the sources from the local directory to the `source` cache directory. It makes `scm_folder.txt` not needed (@niosHD)
- The `scm` auto won't replace the revision if the repo is not pristine. `auto` will be kept.
- The `conan upload` command won't allow uploading a recipe with `revision=auto`.
- The `conan install` will work out of the box, because if the `source` folder exists, it will just use it, without trying to use changes from the local directory.
- The `conan export` and `conan create` will accept a `--scm-dirty` parameter to force capturing the revision (commit) even if the repo is not pristine. Why? if a CI build clone the repository, it is common that some file is added or modified because of the pipeline, but indeed you want to force that commit to be considered valid, so you can write your CI script with the `--scm-dirty` because you know what are you doing.


Feedback, please! Let's target `1.17` for this. @jgsogo feedback welcome.


Discussion:
I would only postpone the `--scm-dirty`, is there anyone requesting it? At first, I don't see the use case, you clone locally and then build in the cache... I don't expect any file _appearing_ in the local folder.

But I like everything else, I think it covers both use cases without introducing any new _magic keyword_ and it is as safer as it was (a little bit more).
Not at local, but in the CI, it is very typical to write in the same folder some log file, some tmp file, some json output...that's why we haven't checked so far the "pristine" state, 
When I said _locally_ I meant _"in a local folder in the CI machine"_, so you do:

```
git clone https://..../mypkg.git
conan create mypkg/conanfile.py --profile=ci-profile
```

and I would expect those "log", "tmp", "json" to be stored next to the cloned repo, but not inside. Of course this could be an scenario, but I think it should be considered a bad practice and, at least, we should wait before implementing it until someone request it vehemently.

----
We didn't implement the _block not-pristine_ because the place where we could do it is in the `conan export ..` command for the local workflow (user computer) and it would block the user there. Now that we are exporting the recipe with `scm="auto"` it is no longer an issue for CI, although it is still for the local-workflow and we will never be able to block the _upload not-pristine_.
Disagree 
> Not at local, but in the CI, it is very typical to write in the same folder some log file, some tmp file, some json output...that's why we haven't checked so far the "pristine" state,

We added any build time generated files to `gitignore` to keep it pristine.
We've noticed for example that Jenkins will generate a `@tmp` directory when using `dir{}` pipeline step. So we've added that to our `gitignore` for example.
I agree that failing when dirty would be a correct decision on your side. But also I know that not all the CI scripts (including some of them cooked by me) are excluding the "CI-non-expected-changes" at the `.gitignore` and probably I don't care much about them. 
Yea, I like the addition of --scm_dirty as it still allows the behavior as desired. :+1: from me on this one!
Sounds good! Having something like `--scm_dirty` is definitely a nice idea too. Having temporary files in the repository on the CI server is very common here. We use Gitlab CI that clones the repository automatically and executes the job within the cloned folder. It is therefore quite easy to end up in a non pristine scm state although nothing important has been changed.

By the way, not a strong feeling but I would use a dash instead of the underscore (i.e., `--scm-dirty`) to be consistent with other long options (e.g., `--package-folder`).
Sure, updated comments to `--scm-dirty`, thanks guys for the feedback!! :heart: 
Just to make sure I understand the differences between `conan install` and `conan create`
If `revision: "auto"` is exported as part of a non-pristine build of a downstream dependency, and you run `--build X` or `--build outdated` on a higher level dependency, is there difference in behavior between `install` and `create`? `create` would use the local source folder, but `install` would not?

I think I just need a bit more clarity on this.
Both will use the `source` folder. I think they will act exactly the same, the [issue](https://github.com/conan-io/conan/issues/4218) about modifying the local directory and getting unexpected code doesn't have anymore because the sources are "frozen" in the `source`
> The conan upload command won't allow uploading a recipe with revision=auto.

Would we want to be able to override to that on specific basis? We use a `depends.json` file for human readable `requirements()` that gets read by the `conanfile.py`
This is checked alongside the source code. Sometimes our automation may explicitly change it to run specific sets of `requirements()`, and we may explicitly want to upload a "dirty build" for testing purposes. These packages are named differently than our official builds, usually relating to CI for a pull request. Our automation pipeline would build then hand off the "dirty" package name/ID for a different pipeline to grab from Artifactory and run tests against.

It isn't **necessary** to provide this behavior, as we already have an override behavior with if another file `dev_depends.json` is detected that is gitignored and will override `depends.json`.

But just a thought. Some users may want to upload these dirty packages as "deployables" for testing. You obviously wouldn't want to rebuild this, as it's a custom version. So I like that it still retains "auto" in the `conanfile.py`
Maybe the current `--force` of the `conan upload` might be able to do that.
Hi all! 
A brief update. I'm trying to implement this but I faced an unexpected issue, the recipes with `source()` method and `scm` with auto, capture the local directory in the source folder during the export, later, the source() doesn't run because there is already a source folder. (we cannot break this optimization either).

I'm not sure how is the best way to solve this, but I would say we would need a different `sources` folder for capturing the local scm, but it has also some implications.

>source() method and scm with auto

We haven't mixed these two solutions - for my general curiosity what sort of use case is there for having both a source() method and scm feature? Do you have an example?
Apply a patch maybe.
> capture the local directory in the source folder during the export, later, the source() doesn't run because there is already a source folder. (we cannot break this optimization either).

Just working off your example - Wouldn't the patch have been run at least once, since the source() would have run the first time before the folder existed?
No, the `source()` method is never called, because of the `source` folder exists.
> A brief update. I'm trying to implement this but I faced an unexpected issue, the recipes with `source()` method and `scm` with auto, capture the local directory in the source folder during the export, later, the source() doesn't run because there is already a source folder. (we cannot break this optimization either).

Did not know that this was possible either. Isn't this a problem for the current `conan create` approach as well? The simplest workaround I can think of is to disable the copying when a custom source method is present in the recipe.
This has been sitting for a while @lasote.
The original suggestion https://github.com/conan-io/conan/issues/5126 that this got wrapped into - is preventing me from rolling out Conan to the wider organization I work for. It is a frequent use-case for our developers to make local changes at different levels of the dependency tree.

Any way we can get some traction on this? 
Hi @Mark-Hatch-Bose 
I'll discuss again with the team to see how we solve the issues with https://github.com/conan-io/conan/pull/5543
I've open a PR and assigned for coming Conan 1.20: https://github.com/conan-io/conan/pull/5543 
Merged. To be released at 1.20
Seeing some unexpected behavior here with `conan v1.20.1` @lasote 

- Make local changes
> git status : modified `blah.sh`
- Run `conan create`, export package `Package/3.9.12+g812a527` (dirty package)
- Everything works fine.

- Make another change, again modify `blah.sh`
> git status : modified blah.sh
- Run `conan create` again
```
Package/3.9.12+g812a527@mark/local: WARN: There are uncommitted changes, skipping the replacement of 'scm.url' and 'scm.revision' auto fields. Use --ignore-dirty to force it. The 'conan upload' command will prevent uploading recipes with 'auto' values in these fields.
Package/3.9.12+g812a527@mark/local: SCM: Getting sources from folder: /scratch/Package
Package/3.9.12+g812a527@mark/local: The stored package has not changed
Package/3.9.12+g812a527@mark/local: Using the exported files summary hash as the recipe revision: 37414aa71d381a86232b7a360f53ea82 
Package/3.9.12+g812a527@mark/local: Exported revision: 37414aa71d381a86232b7a360f53ea82
Package/3.9.12+g812a527@mark/local: Forced build from source
```
All looks good - says it's getting sources from folder again. I expected dirty package.
```
Package/3.9.12+g812a527@mark/local: WARN: Build folder is dirty, removing it: /home/mark/.conan/data/Package/3.9.12+g812a527/mark/local/build/0f2f43134d2a86425d1f56779681a6bd060d600c
Package/3.9.12+g812a527@mark/local: Copying sources to build folder
Package/3.9.12+g812a527@mark/local: Building your package in /home/mark/.conan/data/Package/3.9.12+g812a527/mark/local/build/0f2f43134d2a86425d1f56779681a6bd060d600c
Package/3.9.12+g812a527@mark/local: Generator txt created conanbuildinfo.txt
Package/3.9.12+g812a527@mark/local: Calling build()
```
Great says build folder is being removed - and rebuilding...

However, local changes made 2nd time are not reflected.
Like source was never copied for 2nd time. Is that intentional?
From the prints, and what I would expect - basically the source has to be copied each time.

Doing a `conan remove Package/3.9.12+g812a527@mark/local -f` and then rerunning `create` works fine. It appears the source is not being copied again as indicated in:
>  SCM: Getting sources from folder: /scratch/Package
Thanks for reporting we will take a look
Thanks!
If it helps - I have confirmed that with conan `1.19.4` back to back `create` of same package copies local source again. Slightly different print output below.

```
Package/3.9.12+g812a527@mark/local: WARN: Detected 'scm' auto in conanfile, trying to remove source folder
Package/3.9.12+g812a527@mark/local: WARN: This can take a while for big packages
Package/3.9.12+g812a527@mark/local: Configuring sources in /home/mark/.conan/data/Package/3.9.12+g812a527/mark/local/source
```

Resolving patch:
diff --git a/conans/client/cmd/export.py b/conans/client/cmd/export.py
--- a/conans/client/cmd/export.py
+++ b/conans/client/cmd/export.py
@@ -127,12 +127,17 @@ def cmd_export(app, conanfile_path, name, version, user, channel, keep_source,
                                                               package_layout.export(), output,
                                                               ignore_dirty)
         # Clear previous scm_folder
+        modified_recipe = False
         scm_sources_folder = package_layout.scm_sources()
         rmdir(scm_sources_folder)
         if local_src_folder and not keep_source:
             # Copy the local scm folder to scm_sources in the cache
             mkdir(scm_sources_folder)
             _export_scm(scm_data, local_src_folder, scm_sources_folder, output)
+            # https://github.com/conan-io/conan/issues/5195#issuecomment-551840597
+            # It will cause the source folder to be removed (needed because the recipe still has
+            # the "auto" with uncommitted changes)
+            modified_recipe = True
 
         # Execute post-export hook before computing the digest
         hook_manager.execute("post_export", conanfile=conanfile, reference=package_layout.ref,
@@ -140,7 +145,7 @@ def cmd_export(app, conanfile_path, name, version, user, channel, keep_source,
 
         # Compute the new digest
         manifest = FileTreeManifest.create(package_layout.export(), package_layout.export_sources())
-        modified_recipe = not previous_manifest or previous_manifest != manifest
+        modified_recipe |= not previous_manifest or previous_manifest != manifest
         if modified_recipe:
             output.success('A new %s version was exported' % CONANFILE)
             output.info('Folder: %s' % package_layout.export())


Test patch:
diff --git a/conans/test/functional/scm/scm_test.py b/conans/test/functional/scm/scm_test.py
--- a/conans/test/functional/scm/scm_test.py
+++ b/conans/test/functional/scm/scm_test.py
@@ -11,7 +11,7 @@
 from conans.model.scm import SCMData
 from conans.test.utils.test_files import temp_folder
 from conans.test.utils.tools import NO_SETTINGS_PACKAGE_ID, SVNLocalRepoTestCase, TestClient, \
-    TestServer, create_local_git_repo
+    TestServer, create_local_git_repo, GenConanfile
 from conans.util.files import load, rmdir, save, to_file_bytes
 
 base = '''
@@ -953,6 +953,32 @@ def test_upload_blocking_auto(self):
         self.client.run("upload lib/0.1@user/channel -r default --force")
         self.assertIn("Uploaded conan recipe", self.client.out)
 
+    def test_double_create(self):
+        # https://github.com/conan-io/conan/issues/5195#issuecomment-551848955
+        self.client = TestClient(default_server_user=True)
+        conanfile = str(GenConanfile().
+                        with_scm({"type": "git", "revision": "auto", "url": "auto"}).
+                        with_import("import os").with_import("from conans import tools").
+                        with_name("lib").
+                        with_version("1.0"))
+        conanfile += """
+    def build(self):
+        contents = tools.load("bla.sh")
+        self.output.warn("Bla? {}".format(contents))
+        """
+        self.client.save({"conanfile.py": conanfile, "myfile.txt": "My file is copied"})
+        create_local_git_repo(folder=self.client.current_folder)
+        self.client.run_command('git remote add origin https://myrepo.com.git')
+        #  modified blah.sh
+        self.client.save({"bla.sh": "bla bla"})
+        self.client.run("create . user/channel")
+        self.assertIn("Bla? bla bla", self.client.out)
+        #  modified blah.sh again
+        self.client.save({"bla.sh": "bla2 bla2"})
+        # Run conan create again
+        self.client.run("create . user/channel")
+        self.assertIn("Bla? bla2 bla2", self.client.out)
+
 
 @attr('svn')
 class SCMSVNWithLockedFilesTest(SVNLocalRepoTestCase):