Repository: mlflow/mlflow

Issue:
[BUG] running an Mlflow Project from github URL with 'main' branch instead of 'master'
Thank you for submitting an issue. Please refer to our [issue policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md) for additional information about bug reports. For help with debugging your code, please refer to [Stack Overflow](https://stackoverflow.com/questions/tagged/mlflow).

**Please fill in this bug report template to ensure a timely and thorough response.**

### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [ ] Yes. I can contribute a fix for this bug independently.
- [x] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
- [ ] No. I cannot contribute a bug fix at this time.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: Yes
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS Monterey 12.2.1
- **MLflow installed from (source or binary)**: `pip install mlflow`
- **MLflow version (run ``mlflow --version``)**: 1.25.1
- **Python version**: 3.8.12
- **npm version, if running the dev UI**: -
- **Exact command to reproduce**: `mlflow run https://github.com/adin786/mlflow-test.git`

### Describe the problem
Describe the problem clearly here. Include descriptions of the expected behavior and the actual behavior.

I have written an Mlflow Project specified by an `MLproject` file, in a Github hosted repo.  When creating this repo, Github defaults to calling the starting branch **'origin/main'** rather than the old name **'origin/master'**.

This causes an issue for mlflow.  When I run `mlflow run https://github.com/adin786/mlflow-test.git` without specifying a branch name directly, it fails with the following error message 

```
2022/04/18 22:49:00 INFO mlflow.projects.utils: === Fetching project from https://github.com/adin786/mlflow-test.git into /var/folders/n6/8q8ymx711f981rrq8_6rqnj80000gp/T/tmposbzzzxi ===
Traceback (most recent call last):
  File "/Users/azam/miniforge3/envs/mlflow-test/bin/mlflow", line 8, in <module>
    sys.exit(cli())
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/mlflow/cli.py", line 178, in run
    projects.run(
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/mlflow/projects/__init__.py", line 329, in run
    submitted_run_obj = _run(
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/mlflow/projects/__init__.py", line 95, in _run
    submitted_run = backend.run(
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/mlflow/projects/backend/local.py", line 46, in run
    work_dir = fetch_and_validate_project(project_uri, version, entry_point, params)
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/mlflow/projects/utils.py", line 130, in fetch_and_validate_project
    work_dir = _fetch_project(uri=uri, version=version)
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/mlflow/projects/utils.py", line 164, in _fetch_project
    _fetch_git_repo(parsed_uri, version, dst_dir)
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/mlflow/projects/utils.py", line 202, in _fetch_git_repo
    repo.create_head("master", origin.refs.master)
  File "/Users/azam/miniforge3/envs/mlflow-test/lib/python3.8/site-packages/git/util.py", line 1001, in __getattr__
    return list.__getattribute__(self, attr)
AttributeError: 'IterableList' object has no attribute 'origin/master'
```

It looks like the module `mlflow.projects.utils` contains a call to `repo.create_head("master", origin.refs.master)` if the `--version` flag is not specified.  So mlflow is assuming there is a **'master'** branch and fails because my repo only contains **'main'**.

I found that the issue does not occur if I do one of the following:
- I specify `--version` with a commit hash like `cf576c14af64e4814aa31e7e524c1a6d9f024266`
- I specify `--version` with `main`

**Suggestion:** First check if `master` or `main` exist as branches.

### Code to reproduce issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.

Create a github repo with the following 3 files:

**MLproject** file:
```yaml
name: project_test

conda_env: conda.yaml

entry_points: 
  main:
    command: "python train.py"
```

**conda.yaml** file:
```yaml
name: mlflow-test
channels:
  - conda-forge
dependencies:
  - pip=22.0.4
  - python=3.8.12
  - pip:
    - mlflow==1.25.1
```

**train.py** file:
```python
import mlflow

with mlflow.start_run():
    mlflow.log_param('my_param', 99)
    mlflow.log_metric('my_metric', 55555)
```

Then run:
```bash
git add .
git commit -m "commit message"
git push
mlflow run <github-repo-url>
```

### Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

[See above]

### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [x] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations



Discussion:
I propose if without `--version` specified, automatically detect the "main" or "master" branch, if both of them exists, then it is ambiguous and raise error, but if only one of them exists, then use it.
@WeichenXu123 That proposal sounds great!

@adin786 Feel free to file a pull request; we'd be happy to review it and provide guidance where necessary! Thank you in advance for your contribution!
I tested the change below and it worked:

```diff
diff --git a/mlflow/projects/utils.py b/mlflow/projects/utils.py
index faf250488..f637df2b2 100644
--- a/mlflow/projects/utils.py
+++ b/mlflow/projects/utils.py
@@ -200,8 +200,8 @@ def _fetch_git_repo(uri, version, dst_dir):
             )
     else:
         origin.fetch(depth=GIT_FETCH_DEPTH)
-        repo.create_head("master", origin.refs.master)
-        repo.heads.master.checkout()
+        repo.create_head("default", origin.refs[0])
+        repo.heads.default.checkout()
     repo.submodule_update(init=True, recursive=True)
```

## How this works

1. `origin.fetch(depth=GIT_FETCH_DEPTH)` fethces the default branch. It might be "master", "main", "foo", "bar", or something else. We can just let `gitpython` determine that.
2. `origin.refs[0]` refers to the default branch so set it to "default".
3. Done.

I'm not 100% confident this works so we need to carefully test it.

## How can we test this?

We can test this by creating a few MLflow project repositories with different default branches (e.g. "master", "main", and "foo") and run `mlflow run` against them to make sure the default branch is fetched regardless of its name.

I also tested this but it didn't work:

```diff
diff --git a/mlflow/projects/utils.py b/mlflow/projects/utils.py
index faf250488..8124a3366 100644
--- a/mlflow/projects/utils.py
+++ b/mlflow/projects/utils.py
@@ -200,8 +200,6 @@ def _fetch_git_repo(uri, version, dst_dir):
             )
     else:
         origin.fetch(depth=GIT_FETCH_DEPTH)
-        repo.create_head("master", origin.refs.master)
-        repo.heads.master.checkout()
     repo.submodule_update(init=True, recursive=True)
```

Result

```
$ mlflow run https://github.com/rcallagh/mlflow_submodule_test.git
2022/04/20 11:13:53 INFO mlflow.projects.utils: === Fetching project from https://github.com/rcallagh/mlflow_submodule_test.git into /var/folders/33/kkm875tn7cxb89b8mmcykrx00000gn/T/tmps5qmo1k9 ===
Traceback (most recent call last):
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/bin/mlflow", line 33, in <module>
    sys.exit(load_entry_point('mlflow', 'console_scripts', 'mlflow')())
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/harutakakawamura/Desktop/repositories/mlflow/mlflow/cli.py", line 192, in run
    run_name=run_name,
  File "/Users/harutakakawamura/Desktop/repositories/mlflow/mlflow/projects/__init__.py", line 341, in run
    run_name=run_name,
  File "/Users/harutakakawamura/Desktop/repositories/mlflow/mlflow/projects/__init__.py", line 102, in _run
    experiment_id,
  File "/Users/harutakakawamura/Desktop/repositories/mlflow/mlflow/projects/backend/local.py", line 46, in run
    work_dir = fetch_and_validate_project(project_uri, version, entry_point, params)
  File "/Users/harutakakawamura/Desktop/repositories/mlflow/mlflow/projects/utils.py", line 130, in fetch_and_validate_project
    work_dir = _fetch_project(uri=uri, version=version)
  File "/Users/harutakakawamura/Desktop/repositories/mlflow/mlflow/projects/utils.py", line 164, in _fetch_project
    _fetch_git_repo(parsed_uri, version, dst_dir)
  File "/Users/harutakakawamura/Desktop/repositories/mlflow/mlflow/projects/utils.py", line 203, in _fetch_git_repo
    repo.submodule_update(init=True, recursive=True)
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/git/repo/base.py", line 393, in submodule_update
    return RootModule(self).update(*args, **kwargs)
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/git/objects/submodule/root.py", line 53, in __init__
    parent_commit=repo.head.commit,
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/git/refs/symbolic.py", line 197, in _get_commit
    obj = self._get_object()
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/git/refs/symbolic.py", line 190, in _get_object
    return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/git/refs/symbolic.py", line 132, in dereference_recursive
    hexsha, ref_path = cls._get_ref_info(repo, ref_path)
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/git/refs/symbolic.py", line 181, in _get_ref_info
    return cls._get_ref_info_helper(repo, ref_path)
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/git/refs/symbolic.py", line 164, in _get_ref_info_helper
    raise ValueError("Reference at %r does not exist" % ref_path)
ValueError: Reference at 'refs/heads/master' does not exist
```
I tried the fix mentioned above, i.e. using repo.heads.default.checkout(). It worked for me with the following cases:

1. github public repo that does not require auth at https://github.com/jagane-infinstor/mlflow-example-docker.git 
2. AWS CodeCommit private repo that requires auth

I think this is a better solution that my proposal to try and check out a branch called 'main' if the attempt to checkout a branch called 'master' fails.
@jagane-infinstor Thanks for the comment. Can you update #5872?

Resolving patch:
diff --git a/mlflow/projects/utils.py b/mlflow/projects/utils.py
--- a/mlflow/projects/utils.py
+++ b/mlflow/projects/utils.py
@@ -4,7 +4,6 @@
 import tempfile
 import urllib.parse
 
-
 from distutils import dir_util
 
 import mlflow.utils
@@ -175,6 +174,16 @@ def _unzip_repo(zip_file, dst_dir):
         zip_in.extractall(dst_dir)
 
 
+_HEAD_BRANCH_REGEX = re.compile(r"^\s*HEAD branch:\s+(?P<branch>\S+)")
+
+
+def _get_head_branch(remote_show_output):
+    for line in remote_show_output.splitlines():
+        match = _HEAD_BRANCH_REGEX.match(line)
+        if match:
+            return match.group("branch")
+
+
 def _fetch_git_repo(uri, version, dst_dir):
     """
     Clone the git repo at ``uri`` into ``dst_dir``, checking out commit ``version`` (or defaulting
@@ -199,9 +208,21 @@ def _fetch_git_repo(uri, version, dst_dir):
                 "Error: %s" % (version, uri, e)
             )
     else:
-        origin.fetch(depth=GIT_FETCH_DEPTH)
-        repo.create_head("master", origin.refs.master)
-        repo.heads.master.checkout()
+        g = git.cmd.Git(dst_dir)
+        cmd = ["git", "remote", "show", "origin"]
+        output = g.execute(cmd)
+        head_branch = _get_head_branch(output)
+        if head_branch is None:
+            raise ExecutionException(
+                "Failed to find HEAD branch. Output of `{cmd}`:\n{output}".format(
+                    cmd=" ".join(cmd), output=output
+                )
+            )
+        origin.fetch(head_branch, depth=GIT_FETCH_DEPTH)
+        ref = origin.refs[0]
+        _logger.info("Fetched '%s' branch", head_branch)
+        repo.create_head(head_branch, ref)
+        repo.heads[head_branch].checkout()
     repo.submodule_update(init=True, recursive=True)
 
 


Test patch: