Repository: conan-io/conan

Issue:
Support optional conf-based arguments for controlling dirty repo detection
### What is your suggestion?

Hi there,

Under Conan 2.0 we have been coming against an issue when attempting to use the `scm` and `scm_folder` revision modes on our private Conan project recipes. To cut a long story short, we are often required to have a particular file located alongside the `conanfile.py`; this file is both tracked by git (it cannot be ignored as it must be stored in the repo) and is modified during our CI build processes prior to the Conan package build. As such, when we attempt to either `conan create` or `conan export` etc. we receive the error about the repo being dirty and the command fails. 

I should point out that this file in no way contributes to the build outputs of the Conan package, so it being dirty has no effect on the built package.

I realise that #14330 has gone some of the way to allowing dirty content to exist when using `scm_folder` revisions, but in our specific case it still doesn't provide a solution due to the modified file being located in the same folder as the recipe. It is simply not viable for us to re-organise the structure of our repos so that we could use the functionality in this form.

Would a possible solution to this scenario be to provide an optional config setting that could allow users to provide extra arguments to the `git status -s` command used in the revision calculcation logic, so that specific files/patterns can be excluded from the dirty check e.g. `tools.git:is_dirty_args=':!<file_to_exclude>'` (see https://stackoverflow.com/a/67815271)

It would also be good to have the ability to pass such extra arguments via an optional arg for the `is_dirty` method, and the `get_url_and_commit` method that calls it, within the `conan.tools.scm.Git` tool.

I look forward to your feedback!

### Have you read the CONTRIBUTING guide?

- [X] I've read the CONTRIBUTING guide


Discussion:
Hi @santhonisz 

Thanks for your suggestion. I understand the use case, and it seems enough to be considered.

If this is like something permanent, and you point that you want that in the recipe too for the ``Git()`` helper, wouldn't it make sense to have it as a recipe attribute so it applies always to both things, and not require a ``conf``? Like ``scm_excludes = ...``? (just thinking out loud, and trying to understand the angles).
Hi @memsharded ,

Thanks for the prompt reply. 

Actually your suggestion probably makes more sense. We are using a custom base class for the majority of our Conan recipes where we follow the approach outlined in [Capturing git scm information](https://docs.conan.io/2/examples/tools/scm/git/capture_scm/git_capture_scm.html), so having a recipe attribute that was automatically considered would make it very easy for us to apply it broadly. 

Thinking out loud myself, I wonder if rather than just configuring exclusions, there would be any appetite for an `scm_sources` attribute that behaved like the `export_sources` attribute; allowing both inclusion _and_ exclusion patterns to be set. The dirty repo detection logic could then apply those if the attribute was present on the recipe, or continue to use the current logic if it is not? We don't need the functionality to be this advanced ourselves; the `scm_excludes` you proposed would satisfy out requirements, just trying to think more broadly.


After comments from https://github.com/conan-io/conan/issues/14715#issuecomment-1715228602, adding this for next 2.0.12 discussion and consideration.
Hi everybody,
we have a similar use case:
into ours CI we have the need to build the same artifacts with both conan v1 and conan v2; the CI process add files and modify template files to add build info.

The better solution for our use case is to add an option to export-pkg command and eventually other commands involved to allow scm revision_mode even on dirty repository, like conan v1 does.

We use Gitlab as source repository and CI manager and Artifactory to store the result.
The CI processes c++, java and shell projects, using a lot of bash and python scripts.
For the c++ projects the conan v1 commands flow still used:
conan install, conan build, conan export-pkg, conan upload,
implemented in different scripts to perform the prepare, check, build and finalize steps of the CI;
migrate to conan create command it's hard to think.
The CI process collects build info sent by e-mail and inserted into the built artifact, postgresql db and Confluence pages.

In this scenario an option to add to the conan commands is easy to implement into the scripts.

The expected result in Artifactory is, like conan v1 commands do:
https://xyz.artifactory.it/artifactory/webapp/#/artifacts/browse/tree/General/group/subgroup/project/version/branch/fd98bd4e834e7400133f21f44e42c7c2e4d9cf70
where fd98bd4e834e7400133f21f44e42c7c2e4d9cf70 is the project's commit hash pushed to GitLab generating the CI pipeline.

@santhonisz, a couple of clarifications, please

> our private Conan project recipes. To cut a long story short, we are often required to have a particular file located alongside the conanfile.py; this file is both tracked by git (it cannot be ignored as it must be stored in the repo) and is modified during our CI build processes prior to the Conan package build. As such, when we attempt to either conan create or conan export etc. we receive the error about the repo being dirty and the command fails.

> I should point out that this file in no way contributes to the build outputs of the Conan package, so it being dirty has no effect on the built package.

This file lives in the user folder, together with the ``conanfile.py``, but is it exported to the Conan cache as part of the recipe? Or it only exist in the user folder in pre-export?

In think that for files that are not exported, then, it might not be necessary to do ``exclusion`` mechanism, but just taking the ``git commit``, and using it as if it were not dirty? Or you are interested in the protection of the "dirty" checking for other potential files?

If the file is actually exported into the cache, that might be more problematic, because even if it is not used (why does it need to be exported then), it will break the "immutability" of package recipes. Maybe this would be a candidate for the new ``metadata files`` feature?  (merged, to be documented and launched soon).
Hi @memsharded ,

The file in question is **not** exported to the Conan cache; it is in fact a `sonar-project.properties` file that configures the SonarQube analysis of the package sources. As previously mentioned, this file is committed to the repo and is then dynamically updated as part of our CI build process, hence making the repo "dirty".

It's an interesting point you make about just taking the `git commit`, and using it as if it were not dirty. While I think that would be an acceptable solution for us in terms of CI builds, I do like the protection the current "dirty" checking provides for developers who are running `conan create/export` locally e.g. if they have uncommitted source changes in their user folder, bypassing the dirty check would lead to inconsistent results when running `conan create/export` vs `conan build`.

To your last point, I completely agree and we have no such scenario where a "dirty" file is exported because, as you say, that would break the "immutability" of the package.
Hello,

Protecting from dirty sources and keeping immutability makes total sense. Nevertheless, isn't there a bug in the [`is_dirty`](https://github.com/conan-io/conan/blob/release/2.0/conan/tools/scm/git.py#L89) method? It is mentionned that the **current folder** is checked, but the executed Git command is `git status -s`. This will check the whole repository for dirty files. The command is missing the folder option: `git status -s -- "."`. As a consequnce, it is currently not possible to call `get_url_and_commit` as soon as there is a modification in the repository, even in a different folder than the recipe.
> Nevertheless, isn't there a bug in the [is_dirty](https://github.com/conan-io/conan/blob/release/2.0/conan/tools/scm/git.py#L89) method? It is mentionned that the current folder is checked, but the executed Git command is git status -s. This will check the whole repository for dirty files. The command is missing the folder option: git status -s -- ".". As a consequnce, it is currently not possible to call get_url_and_commit as soon as there is a modification in the repository, even in a different folder than the recipe.

Thanks for pointing this out. Yes, I think it is worth checking this, lets have a look
I am doing https://github.com/conan-io/conan/pull/15289 for fixing the bug of ``Git.is_dirty()``, sorry for the delays, it has been quite busy, I am trying to revive this feature request, but the bugfix has higher priority.
By the way, I am checking possible implementations, and it seems ``git`` already has some mechanisms to define this like:

```
# do not track changes
git update-index --assume-unchanged path/to/file

# continue tracking changes
git update-index --no-assume-unchanged path/to/file
```

At least that is a potential workaround that CI can define to avoid the Conan recipe failing because of dirty?

No worries on the delays, appreciate there's been more pressing issues!

To your comment, we've in fact been doing something similar in CI to date, though with `--skip-worktree/--no-skip-worktree` in place of `--assume-unchanged/--no-assume-unchanged`. It is workable, though we'd obviously like to see a cleaner/less-impactful solution in place such as passing the aforementioned exclusion patterns to the `git status` command.

I am trying to find a way to have ``git`` be the one that executes the exclusion. Because we did it in the past, and it was not that evident, a bit too complicated and fragile.

I am thinking about trying to add something to .gitignore temporarily or some other similar git trick, like ``--assume-unchanged``, passing the user ``excluded`` argument, but without having to filter ourselves the changeset.
https://github.com/conan-io/conan/pull/15289 is doing the fix for the path ``is_dirty()`` for 2.0.15, will try the exclude for next release
I have tried several git built-in approaches, but I didn't find anything effective

I am trying in https://github.com/conan-io/conan/pull/15457 a ``git_excluded = ["myfile*.txt", ...]`` fnmatch based approach, that works both for tracked and untracked files modifications. Feedback and testing very welcome.



Patch:
diff --git a/conan/tools/scm/git.py b/conan/tools/scm/git.py
index 8ba755b8193..77d0e6dbbfd 100644
--- a/conan/tools/scm/git.py
+++ b/conan/tools/scm/git.py
@@ -1,7 +1,9 @@
+import fnmatch
 import os
 
 from conan.tools.files import chdir
 from conan.errors import ConanException
+from conans.model.conf import ConfDefinition
 from conans.util.files import mkdir
 from conans.util.runners import check_output_runner
 
@@ -10,13 +12,24 @@ class Git:
     """
     Git is a wrapper for several common patterns used with *git* tool.
     """
-    def __init__(self, conanfile, folder="."):
+    def __init__(self, conanfile, folder=".", excluded=None):
         """
         :param conanfile: Conanfile instance.
         :param folder: Current directory, by default ``.``, the current working directory.
         """
         self._conanfile = conanfile
         self.folder = folder
+        self._excluded = excluded
+        global_conf = conanfile._conan_helpers.global_conf
+        conf_excluded = global_conf.get("core.scm:excluded", check_type=list)
+        if conf_excluded:
+            if excluded:
+                c = ConfDefinition()
+                c.loads(f"core.scm:excluded={excluded}")
+                c.update_conf_definition(global_conf)
+                self._excluded = c.get("core.scm:excluded", check_type=list)
+            else:
+                self._excluded = conf_excluded
 
     def run(self, cmd):
         """
@@ -107,7 +120,15 @@ def is_dirty(self):
         :return: True, if the current folder is dirty. Otherwise, False.
         """
         status = self.run("status . --short --no-branch --untracked-files").strip()
-        return bool(status)
+        self._conanfile.output.debug(f"Git status:\n{status}")
+        if not self._excluded:
+            return bool(status)
+        # Parse the status output, line by line, and match it with "_excluded"
+        lines = [line.strip() for line in status.splitlines()]
+        lines = [line.split()[1] for line in lines if line]
+        lines = [line for line in lines if not any(fnmatch.fnmatch(line, p) for p in self._excluded)]
+        self._conanfile.output.debug(f"Filtered git status: {lines}")
+        return bool(lines)
 
     def get_url_and_commit(self, remote="origin", repository=False):
         """
diff --git a/conans/client/cmd/export.py b/conans/client/cmd/export.py
index 660ab924668..afdbd087811 100644
--- a/conans/client/cmd/export.py
+++ b/conans/client/cmd/export.py
@@ -3,12 +3,12 @@
 
 from conan.tools.files import copy
 from conan.api.output import ConanOutput
+from conan.tools.scm import Git
 from conans.errors import ConanException, conanfile_exception_formatter
 from conans.model.manifest import FileTreeManifest
 from conans.model.recipe_ref import RecipeReference
 from conans.paths import DATA_YML
 from conans.util.files import is_dirty, rmdir, set_dirty, mkdir, clean_dirty, chdir
-from conans.util.runners import check_output_runner
 
 
 def cmd_export(app, global_conf, conanfile_path, name, version, user, channel, graph_lock=None,
@@ -61,7 +61,8 @@ def cmd_export(app, global_conf, conanfile_path, name, version, user, channel, g
     revision = _calc_revision(scoped_output=conanfile.output,
                               path=os.path.dirname(conanfile_path),
                               manifest=manifest,
-                              revision_mode=conanfile.revision_mode)
+                              revision_mode=conanfile.revision_mode,
+                              conanfile=conanfile)
 
     ref.revision = revision
     recipe_layout.reference = ref
@@ -86,7 +87,7 @@ def cmd_export(app, global_conf, conanfile_path, name, version, user, channel, g
     return ref, conanfile
 
 
-def _calc_revision(scoped_output, path, manifest, revision_mode):
+def _calc_revision(scoped_output, path, manifest, revision_mode, conanfile):
     if revision_mode not in ["scm", "scm_folder", "hash"]:
         raise ConanException("Revision mode should be one of 'hash' (default) or 'scm'")
 
@@ -94,19 +95,20 @@ def _calc_revision(scoped_output, path, manifest, revision_mode):
     if revision_mode == "hash":
         revision = manifest.summary_hash
     else:
-        f = '-- "."' if revision_mode == "scm_folder" else ""
+        # Exception to the rule that tools should only be used in recipes, this Git helper is ok
+        excluded = getattr(conanfile, "revision_mode_excluded", None)
+        git = Git(conanfile, folder=path, excluded=excluded)
         try:
-            with chdir(path):
-                revision = check_output_runner(f'git rev-list HEAD -n 1 --full-history {f}').strip()
+            revision = git.get_commit(repository=(revision_mode == "scm"))
         except Exception as exc:
             error_msg = "Cannot detect revision using '{}' mode from repository at " \
                         "'{}'".format(revision_mode, path)
             raise ConanException("{}: {}".format(error_msg, exc))
 
-        with chdir(path):
-            if bool(check_output_runner(f'git status -s {f}').strip()):
-                raise ConanException("Can't have a dirty repository using revision_mode='scm' and doing"
-                                     " 'conan export', please commit the changes and run again.")
+        if git.is_dirty():
+            raise ConanException("Can't have a dirty repository using revision_mode='scm' and doing"
+                                 " 'conan export', please commit the changes and run again, or "
+                                 "use 'git_excluded = []' attribute")
 
         scoped_output.info("Using git commit as the recipe revision: %s" % revision)
 
diff --git a/conans/model/conf.py b/conans/model/conf.py
index 30e1b169efd..fa34aef855b 100644
--- a/conans/model/conf.py
+++ b/conans/model/conf.py
@@ -48,6 +48,8 @@
     "core.net.http:clean_system_proxy": "If defined, the proxies system env-vars will be discarded",
     # Gzip compression
     "core.gzip:compresslevel": "The Gzip compression level for Conan artifacts (default=9)",
+    # Excluded from revision_mode = "scm" dirty and Git().is_dirty() checks
+    "core.scm:excluded": "List of excluded patterns for builtin git dirty checks",
     # Tools
     "tools.android:ndk_path": "Argument for the CMAKE_ANDROID_NDK",
     "tools.android:cmake_legacy_toolchain": "Define to explicitly pass ANDROID_USE_LEGACY_TOOLCHAIN_FILE in CMake toolchain",


Test patch:
diff --git a/conans/test/functional/command/export_test.py b/conans/test/functional/command/export_test.py
index 806d86c7a38..9fc74db5110 100644
--- a/conans/test/functional/command/export_test.py
+++ b/conans/test/functional/command/export_test.py
@@ -5,6 +5,7 @@
 from conans.test.assets.genconanfile import GenConanfile
 from conans.test.utils.scm import git_add_changes_commit
 from conans.test.utils.tools import TestClient
+from conans.util.files import save
 
 
 @pytest.mark.tool("git")
@@ -60,3 +61,31 @@ def test_auto_revision_without_commits(self):
         t.run("export .", assert_error=True)
         # It errors, because no commits yet
         assert "Cannot detect revision using 'scm' mode from repository" in t.out
+
+    @pytest.mark.parametrize("conf_excluded, recipe_excluded",
+                             [("", ["*.cpp", "*.txt", "src/*"]),
+                              (["*.cpp", "*.txt", "src/*"], ""),
+                              ('+["*.cpp", "*.txt"]', ["src/*"]),
+                              ('+["*.cpp"]', ["*.txt", "src/*"])])
+    def test_revision_mode_scm_excluded_files(self, conf_excluded, recipe_excluded):
+        t = TestClient()
+        recipe_excluded = f'revision_mode_excluded = {recipe_excluded}' if recipe_excluded else ""
+        conf_excluded = f'core.scm:excluded={conf_excluded}' if conf_excluded else ""
+        save(t.cache.global_conf_path, conf_excluded)
+        conanfile = GenConanfile("pkg", "0.1").with_class_attribute('revision_mode = "scm"') \
+                                              .with_class_attribute(recipe_excluded)
+        commit = t.init_git_repo({'conanfile.py': str(conanfile),
+                                  "test.cpp": "mytest"})
+
+        t.run(f"export .")
+        assert t.exported_recipe_revision() == commit
+
+        t.save({"test.cpp": "mytest2",
+                "new.txt": "new",
+                "src/potato": "hello"})
+        t.run(f"export . -vvv")
+        assert t.exported_recipe_revision() == commit
+
+        t.save({"test.py": ""})
+        t.run(f"export .", assert_error=True)
+        assert "ERROR: Can't have a dirty repository using revision_mode='scm'" in t.out
diff --git a/conans/test/functional/tools/scm/test_git.py b/conans/test/functional/tools/scm/test_git.py
index 616e635fb2d..af096464243 100644
--- a/conans/test/functional/tools/scm/test_git.py
+++ b/conans/test/functional/tools/scm/test_git.py
@@ -10,7 +10,7 @@
 from conans.test.utils.scm import create_local_git_repo, git_add_changes_commit, git_create_bare_repo
 from conans.test.utils.test_files import temp_folder
 from conans.test.utils.tools import TestClient
-from conans.util.files import rmdir, save_files
+from conans.util.files import rmdir, save_files, save
 
 
 @pytest.mark.tool("git")
@@ -26,7 +26,7 @@ class Pkg(ConanFile):
             version = "0.1"
 
             def export(self):
-                git = Git(self, self.recipe_folder)
+                git = Git(self, self.recipe_folder, excluded=["myfile.txt", "mynew.txt"])
                 commit = git.get_commit()
                 repo_commit = git.get_commit(repository=True)
                 url = git.get_remote_url()
@@ -116,6 +116,29 @@ def test_capture_commit_local_subfolder(self):
         assert "pkg/0.1: COMMIT IN REMOTE: False" in c.out
         assert "pkg/0.1: DIRTY: False" in c.out
 
+    def test_git_excluded(self):
+        """
+        A local repo, without remote, will have commit, but no URL
+        """
+        c = TestClient()
+        c.save({"conanfile.py": self.conanfile,
+                "myfile.txt": ""})
+        c.init_git_repo()
+        c.run("export . -vvv")
+        assert "pkg/0.1: DIRTY: False" in c.out
+        c.save({"myfile.txt": "changed",
+                "mynew.txt": "new"})
+        c.run("export .")
+        assert "pkg/0.1: DIRTY: False" in c.out
+        c.save({"other.txt": "new"})
+        c.run("export .")
+        assert "pkg/0.1: DIRTY: True" in c.out
+
+        conf_excluded = f'core.scm:excluded+=["other.txt"]'
+        save(c.cache.global_conf_path, conf_excluded)
+        c.run("export .")
+        assert "pkg/0.1: DIRTY: False" in c.out
+
 
 @pytest.mark.tool("git")
 class TestGitCaptureSCM: