Repository: mlflow/mlflow

Issue:
[BUG] Git fails checking out old commit with submodules
Thank you for submitting an issue. Please refer to our [issue policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md) for additional information about bug reports. For help with debugging your code, please refer to [Stack Overflow](https://stackoverflow.com/questions/tagged/mlflow).

**Please fill in this bug report template to ensure a timely and thorough response.**

### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [ ] Yes. I can contribute a fix for this bug independently.
- [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
- [x] No. I cannot contribute a bug fix at this time.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: yes 
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Tested on macOS 12.1 and Ubuntu 20.04
- **MLflow installed from (source or binary)**: pip
- **MLflow version (run ``mlflow --version``)**: 1.24.0
- **Python version**: 3.9.12
- **npm version, if running the dev UI**:
- **Exact command to reproduce**: mlflow run https://github.com/rcallagh/mlflow_submodule_test.git -v 42886f166fd550ebc56925af36ff80fd92d0095b

### Describe the problem
When trying to run mlflow on a git project containing a submodule using a commit that is not the head of the master branch I get an error 

```
2022/04/11 14:01:46 ERROR mlflow.cli: === Unable to checkout version '42886f166fd550ebc56925af36ff80fd92d0095b' of git repo https://github.com/rcallagh/mlflow_submodule_test.git- please ensure that the version exists in the repo. Error: Cmd('git') failed due to: exit code(128)
  cmdline: git checkout 42886f166fd550ebc56925af36ff80fd92d0095b
  stderr: 'fatal: reference is not a tree: 42886f166fd550ebc56925af36ff80fd92d0095b' ===
```

I think that this has something to do with the submodule being present in the repo (as discussed here https://stackoverflow.com/questions/13425298/git-submodule-update-fatal-error-reference-is-not-a-tree) but I'm not sure how to fix it. 

### Code to reproduce issue
I have created a toy example in https://github.com/rcallagh/mlflow_submodule_test which has a submodule https://github.com/rcallagh/submod_for_mlflow_test. If I run `mlflow run https://github.com/rcallagh/mlflow_submodule_test.git -v 4ed55d3d2655a710e0de3887e2166be585a66544` which is the most recent commit on the main branch the it runs successfully. If I run `mlflow run https://github.com/rcallagh/mlflow_submodule_test.git -v 42886f166fd550ebc56925af36ff80fd92d0095b` which is the previous commit, it fails with the error above.  

### Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.


### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [x] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations



Discussion:

Resolving patch:
diff --git a/mlflow/projects/utils.py b/mlflow/projects/utils.py
--- a/mlflow/projects/utils.py
+++ b/mlflow/projects/utils.py
@@ -188,9 +188,9 @@ def _fetch_git_repo(uri, version, dst_dir):
 
     repo = git.Repo.init(dst_dir)
     origin = repo.create_remote("origin", uri)
-    origin.fetch(depth=GIT_FETCH_DEPTH)
     if version is not None:
         try:
+            origin.fetch(refspec=version, depth=GIT_FETCH_DEPTH)
             repo.git.checkout(version)
         except git.exc.GitCommandError as e:
             raise ExecutionException(
@@ -199,6 +199,7 @@ def _fetch_git_repo(uri, version, dst_dir):
                 "Error: %s" % (version, uri, e)
             )
     else:
+        origin.fetch(depth=GIT_FETCH_DEPTH)
         repo.create_head("master", origin.refs.master)
         repo.heads.master.checkout()
     repo.submodule_update(init=True, recursive=True)


Test patch:
diff --git a/tests/projects/test_utils.py b/tests/projects/test_utils.py
--- a/tests/projects/test_utils.py
+++ b/tests/projects/test_utils.py
@@ -2,6 +2,7 @@
 import os
 import tempfile
 
+import requests
 import pytest
 from unittest import mock
 
@@ -105,6 +106,21 @@ def test__fetch_git_repo(local_git_repo, local_git_repo_uri, version, expected_v
     assert repo.active_branch.name == expected_version
 
 
+@pytest.mark.parametrize(
+    "commit",
+    # Fetch the most recent two commits
+    requests.get("https://api.github.com/repos/mlflow/mlflow-example/commits").json()[:2],
+)
+def test_fetch_git_repo_commit(tmp_path, commit):
+    _fetch_git_repo(
+        "https://github.com/mlflow/mlflow-example.git",
+        commit["sha"],
+        tmp_path,
+    )
+    repo = git.Repo(tmp_path)
+    assert repo.commit().hexsha == commit["sha"]
+
+
 def test_fetching_non_existing_version_fails(local_git_repo, local_git_repo_uri):
     with pytest.raises(ExecutionException, match="Unable to checkout"):
         _fetch_git_repo(local_git_repo_uri, "non-version", local_git_repo)