Repository: PrefectHQ/prefect

Issue:
Git Storage is Currently Incompatible with Azure DevOps
## Opened from the [Prefect Public Slack Community](https://prefect.io/slack)

**adam.everington**: Hey all, i'm struggling to connect to our Azure Devops Repo using the GIT storage class. I keep getting an HTTP Unauthorized error. I've tried connecting with git credentials and with a PAT and neither work. Has anyone else managed to connect to AZ Devops?

**kevin701**: Hey <@U02A67U1UUA>, I have not seen Azure DevOps come up here before. I did see <https://prefect-community.slack.com/archives/CL09KU1K7/p1628171528402100|this> which was connections to Azure Storage and Azure Container Registry. I don’t know if it’s a similar format. I can try this and get back to you by Monday.

**adam.everington**: Hey Kevin, thanks that would be great, i'll continue to try in the mean time, I know that the connection to Devops can be awkward at the best of times and in other c# apps we have we've had to request a token and then encode and embed that into the request header.

**kevin701**: Hey man, I have figured out the issue after digging a bit. Basically we form the URL is the params provided and the URL takes this form:
`url = f"https://{self.git_token_secret}@{self.repo_host}/{self.repo}.git"`

This works for Github, Bitbucket, and Gitlab where the Git repo cloning ends in `.git` . For Azure DevOps though, the URL looks something like this:
`working_url = "<https://kdykho@dev.azure.com/kdykho/prefect/_git/prefect>"`

That ending `.git` causes the URL to fail, but azure devops has a different URL.

I’ll chat with an engineer about how to go about this.

**kevin701**: <@ULVA73B9P> open “Git Storage is Currently Incompatible with Azure DevOps”

Original thread can be found [here](https://prefect-community.slack.com/archives/CL09KU1K7/p1628238958463000?thread_ts=1628238958.463000&cid=CL09KU1K7).




Discussion:
To replicate this:

```
url = "https://kdykho:key@dev.azure.com/kdykho/prefect/_git/prefect"

with TemporaryGitRepo(git_clone_url=url):
    print("test")
```

Will pull down the repo but adding any `.git` to the URL will cause the clone to fail. Azure DevOps just has a differently formatted URL. 
I did raise the hard-coded URL template as a possible issue when Git storage first got added in #4418, but there wasn't any clear use cases for allowing custom URLs at the time. Interesting to hear that the `.git` suffix for bare upstream repos isn't used everywhere.

Resolving patch:
diff --git a/src/prefect/serialization/storage.py b/src/prefect/serialization/storage.py
--- a/src/prefect/serialization/storage.py
+++ b/src/prefect/serialization/storage.py
@@ -161,6 +161,7 @@ class Meta:
     flow_name = fields.String(allow_none=True)
     git_token_secret_name = fields.String(allow_none=True)
     git_token_username = fields.String(allow_none=True)
+    git_clone_url_secret_name = fields.String(allow_none=True)
     branch_name = fields.String(allow_none=True)
     tag = fields.String(allow_none=True)
     commit = fields.String(allow_none=True)
diff --git a/src/prefect/storage/git.py b/src/prefect/storage/git.py
--- a/src/prefect/storage/git.py
+++ b/src/prefect/storage/git.py
@@ -42,6 +42,10 @@ class Git(Storage):
             an access token for the repo. Defaults to None
         - git_token_username (str, optional): the username associated with git access token,
             if not provided it will default to repo owner
+        - git_clone_url_secret_name (str, optional): the name of the Prefect Secret specifying
+            the exact git url to clone, if provided it will override repo, repo_host,
+            git_token_secret_name, `git_token_username`, `use_ssh`, and `format_access_token`
+            parameters
         - branch_name (str, optional): branch name, if not specified and `tag` and `commit_sha`
             not specified, repo default branch latest commit will be used
         - tag (str, optional): tag name, if not specified and `branch_name` and `commit_sha`
@@ -64,6 +68,7 @@ def __init__(
         flow_name: str = None,
         git_token_secret_name: str = None,
         git_token_username: str = None,
+        git_clone_url_secret_name: str = None,
         branch_name: str = None,
         tag: str = None,
         commit: str = None,
@@ -83,16 +88,36 @@ def __init__(
                 "SSH will be used to clone the repository. `git_token_secret_name` will be ignored"
             )
 
+        if git_clone_url_secret_name and any(
+            [
+                repo,
+                repo_host,
+                git_token_secret_name,
+                git_token_username,
+                use_ssh,
+                format_access_token,
+            ]
+        ):
+            self.logger.warning(
+                "Git storage initialized with a `git_clone_url_secret_name`. The value of this Secret "
+                "will be used to clone the repository, ignoring `repo`, `repo_host`, "
+                "`git_token_secret_name`,  `git_token_username`, `use_ssh`, and "
+                "`format_access_token`."
+            )
+
         self.flow_path = flow_path
         self.repo = repo
         self.repo_host = repo_host
         self.flow_name = flow_name
         self.git_token_secret_name = git_token_secret_name
+        self.git_clone_url_secret_name = git_clone_url_secret_name
 
         # if not provided, assume the username associated with the token
         # is the organization that owns the repo
         self.git_token_username = (
-            git_token_username if git_token_username else repo.split("/")[0]
+            git_token_username
+            if git_token_username
+            else (repo.split("/")[0] if repo else None)
         )
 
         self.branch_name = branch_name
@@ -166,6 +191,9 @@ def git_clone_url(self) -> str:
         """
         Build the git url to clone
         """
+        if self.git_clone_url_secret_name:
+            return Secret(self.git_clone_url_secret_name).get()  # type: ignore
+
         if self.use_ssh:
             return f"git@{self.repo_host}:{self.repo}.git"
         return f"https://{self.git_token_secret}@{self.repo_host}/{self.repo}.git"


Test patch:
diff --git a/tests/storage/test_git_storage.py b/tests/storage/test_git_storage.py
--- a/tests/storage/test_git_storage.py
+++ b/tests/storage/test_git_storage.py
@@ -1,11 +1,9 @@
 from unittest.mock import MagicMock
-from tempfile import TemporaryDirectory
 
 import pytest
 
 dulwich = pytest.importorskip("dulwich")
 
-import prefect
 from prefect import context, Flow
 from prefect.storage import Git
 
@@ -76,6 +74,21 @@ def test_create_git_storage_with_use_ssh_and_git_token_secret_name_logs_warning(
     )
 
 
+def test_create_git_storage_with_git_clone_url_secret_name_and_other_repo_params_logs_warning(
+    caplog,
+):
+    storage = Git(
+        flow_path="flow.py",
+        repo="test/repo",
+        git_clone_url_secret_name="MY_SECRET",
+    )
+
+    assert (
+        "Git storage initialized with a `git_clone_url_secret_name`. The value of this Secret will be used to clone the repository, ignoring"
+        in caplog.text
+    )
+
+
 def test_create_git_storage_with_tag_and_branch_name_errors():
     with pytest.raises(ValueError):
         storage = Git(
@@ -91,6 +104,7 @@ def test_serialize_git_storage():
         flow_name="my-flow",
         git_token_secret_name="MY_TOKEN",
         git_token_username="my_user",
+        git_clone_url_secret_name="MY_GIT_URL",
         branch_name="my_branch",
         tag=None,
         commit=None,
@@ -107,6 +121,7 @@ def test_serialize_git_storage():
     assert serialized_storage["flow_name"] == "my-flow"
     assert serialized_storage["git_token_secret_name"] == "MY_TOKEN"
     assert serialized_storage["git_token_username"] == "my_user"
+    assert serialized_storage["git_clone_url_secret_name"] == "MY_GIT_URL"
     assert serialized_storage["branch_name"] == "my_branch"
     assert serialized_storage["tag"] == None
     assert serialized_storage["commit"] == None
@@ -137,7 +152,7 @@ def test_add_flow_to_git_already_added():
 
 
 @pytest.mark.parametrize(
-    "repo_host, repo, access_token_secret_name, access_token_secret, access_token_username, use_ssh, format_access_token, expected_git_clone_url",
+    "repo_host, repo, access_token_secret_name, access_token_secret, access_token_username, use_ssh, format_access_token, git_clone_url_secret_name, git_clone_url_secret, expected_git_clone_url",
     [
         # no access token secret provided
         (
@@ -148,6 +163,8 @@ def test_add_flow_to_git_already_added():
             None,
             False,
             True,
+            None,
+            None,
             "https://@github.com/my/repo.git",
         ),
         # github + access token
@@ -159,6 +176,8 @@ def test_add_flow_to_git_already_added():
             None,
             False,
             True,
+            None,
+            None,
             "https://user:MY_SECRET_TOKEN@github.com/user/repo.git",
         ),
         # github + access token + custom token username
@@ -170,6 +189,8 @@ def test_add_flow_to_git_already_added():
             "another_git_user",
             False,
             True,
+            None,
+            None,
             "https://another_git_user:MY_SECRET_TOKEN@github.com/user/repo.git",
         ),
         # bitbucket + access token
@@ -181,6 +202,8 @@ def test_add_flow_to_git_already_added():
             None,
             False,
             True,
+            None,
+            None,
             "https://user:MY_SECRET_TOKEN@bitbucket.org/user/repo.git",
         ),
         # gitlab + access token
@@ -192,6 +215,8 @@ def test_add_flow_to_git_already_added():
             None,
             False,
             True,
+            None,
+            None,
             "https://user:MY_SECRET_TOKEN@gitlab.com/user/repo.git",
         ),
         # unknown hostname + access token
@@ -203,6 +228,8 @@ def test_add_flow_to_git_already_added():
             None,
             False,
             True,
+            None,
+            None,
             "https://user:MY_SECRET_TOKEN@randomsite.com/user/repo.git",
         ),
         # ssh
@@ -214,6 +241,8 @@ def test_add_flow_to_git_already_added():
             None,
             True,  # use ssh instead of https
             True,
+            None,
+            None,
             "git@github.com:user/repo.git",
         ),
         # skip access token formatting
@@ -225,8 +254,23 @@ def test_add_flow_to_git_already_added():
             None,
             False,
             False,  # skip access token formatting
+            None,
+            None,
             "https://MY_SECRET_TOKEN@github.com/user/repo.git",
         ),
+        # provide git clone url secret name directly
+        (
+            None,
+            None,
+            None,
+            None,
+            None,
+            False,
+            True,
+            "GIT_CLONE_URL_SECRET",  # provide the url as a secret
+            "https://my-secret-url.com/repo.git",
+            "https://my-secret-url.com/repo.git",
+        ),
     ],
 )
 def test_get_git_clone_url(
@@ -237,6 +281,8 @@ def test_get_git_clone_url(
     access_token_username,
     use_ssh,
     format_access_token,
+    git_clone_url_secret_name,
+    git_clone_url_secret,
     expected_git_clone_url,
 ):
     storage = Git(
@@ -245,10 +291,16 @@ def test_get_git_clone_url(
         repo_host=repo_host,
         git_token_secret_name=access_token_secret_name,
         git_token_username=access_token_username,
+        git_clone_url_secret_name=git_clone_url_secret_name,
         use_ssh=use_ssh,
         format_access_token=format_access_token,
     )
-    with context(secrets={access_token_secret_name: access_token_secret}):
+    with context(
+        secrets={
+            access_token_secret_name: access_token_secret,
+            git_clone_url_secret_name: git_clone_url_secret,
+        }
+    ):
         assert storage.git_clone_url == expected_git_clone_url