Repository: mesonbuild/meson

Issue:
Regression in wrap support with git
Since https://github.com/mesonbuild/meson/commit/0008b326ffc39c86be2ae4352bdddc03f01840cc#diff-b50022e0b6a45e8e8d96225bdce8f92a we have failures with in CI with the CentOS 7 build with a project that uses a wrap with a git upstream.

```
# git --version
git version 1.8.3.1
```
```
Initialized empty Git repository in /root/toolkit/subprojects/posixsignalmanager/.git/
remote: Counting objects: 68, done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 68 (delta 41), reused 0 (delta 0)
Unpacking objects: 100% (68/68), done.
From git.somewhere.de:posixsignalmanager
 * branch            master     -> FETCH_HEAD
error: pathspec 'master' did not match any file(s) known to git.
```

This is also reproduces with executing the git commands by hand as following:

```
# git init posixsignalmanager
# cd posixsignalmanager/
# git remote add origin git@git.somewhere.de:posixsignalmanager
# git fetch origin master
# git checkout master
error: pathspec 'master' did not match any file(s) known to git.
```

This might be related:
> git branch -a

Does not return anything, so it seems this git version does not create a upstream branch in response to the fetch.

CC @WSoptics 


Discussion:
Does this work: `git checkout -b master origin/master` ? Or `git checkout origin/master` ?
@xclaesse no, unfortunately not

```
# git init posixsignalmanager
Initialized empty Git repository in /posixsignalmanager/.git/
# cd posixsignalmanager/
# git remote add origin git@git.example.de:posixsignalmanager
# git fetch origin master
remote: Counting objects: 68, done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 68 (delta 41), reused 0 (delta 0)
Unpacking objects: 100% (68/68), done.
From git.example.de:posixsignalmanager
 * branch            master     -> FETCH_HEAD
# git checkout -b master origin/master
fatal: Cannot update paths and switch to branch 'master' at the same time.
Did you intend to checkout 'origin/master' which can not be resolved as commit?
```

```
# git init posixsignalmanager
Initialized empty Git repository in /posixsignalmanager/.git/
# cd posixsignalmanager/
# git remote add origin git@git.example.de:posixsignalmanager
# git fetch origin master
remote: Counting objects: 68, done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 68 (delta 41), reused 0 (delta 0)
Unpacking objects: 100% (68/68), done.
From git.example.de:posixsignalmanager
 * branch            master     -> FETCH_HEAD
# git checkout origin/master
error: pathspec 'origin/master' did not match any file(s) known to git.
```
Doing something similar under debian stretch there is a small but likely important difference:
```
$ git fetch origin master
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 3
Unpacking objects: 100% (4/4), done.
From https://github.com/mesonbuild/fmt
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master
```

This has an additional line which shows the creation of 'origin/master'. For some reason this does not happen in the case that fails.
Tested git version:
```
$ git --version
git version 2.11.0
```
The following works for me both with git 2 as well as with git 1.8:

```sh
git init t &&                                                                                                                                     
cd t &&                                                                                                                                           
git remote add origin https://github.com/rtyley/small-test-repo.git &&                                                                            
git fetch origin master &&                                                                                                                        
git checkout -b master FETCH_HEAD &&                                                                                                              
echo 'SUCCESS'                
```

@istoph Can you please check if that fixes your issue and either ideally prepare a pull request that incorporates the `git checkout -b ${branchname} FETCH_HEAD` yourself or alternatively wait for me to do it -- which will be a bit because I am travelling right now and won't have proper time and computer access for a while.
But is "git checkout -b master FETCH_HEAD" the right fix? That might break `vcs_tag` and similar code that allows the (sub)-project to retrieve version information from git.
I would think that we need a fix that does not break this. Does anybody know why git does not add origin/master in this situation? We even might want to keep the old code path for situations where `depth` is not used, to be on the safe side. 
```
# git init t &&                                                                                                                                     
> cd t &&                                                                                                                                           
> git remote add origin https://github.com/rtyley/small-test-repo.git &&                                                                            
> git fetch origin master &&                                                                                                                        
> git checkout -b master FETCH_HEAD &&                                                                                                              
> echo 'SUCCESS'   
Initialized empty Git repository in /t/.git/
remote: Enumerating objects: 6, done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 6
Unpacking objects: 100% (6/6), done.
From https://github.com/rtyley/small-test-repo
 * branch            master     -> FETCH_HEAD
Already on 'master'
SUCCESS
```
this works for me, but i agree with @textshell concern
Do wraps really start with `git init`? That's not the same as `git clone` because it will checkout the branch for you, but an initial repo doesn't have a branch (it has no commits). IOW, are you sure your simplified reproducer is actually equivalent to what wraps do?
Yes, the commit cited in the issue description unconditionally changed the wrap code from using `git clone` to manually emulating git clone with the steps above. I don't know if this is needed for the shallow cloning support added in that patch, but i'm concerned that this will break other use cases as well as it currently breaks centos 7.
Ah yes, I see. Well, `git checkout master` is probably unsafe as it relies on git automatically choosing to link to the `origin` remote, but I'm not sure it always did that. And `git fetch origin master` may or may not create a branch, as shown above. You can do something like (tested in the `centos:7` container):
```
$ git init t
Initialized empty Git repository in /root/t/.git/
$ cd t
$ git remote add origin https://github.com/rtyley/small-test-repo.git
$ git fetch origin master:a-temp-name
remote: Enumerating objects: 6, done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 6
Unpacking objects: 100% (6/6), done.
From https://github.com/rtyley/small-test-repo
 * [new branch]      master     -> a-temp-name
$ git merge a-temp-name
```
_Maybe_ `FETCH_HEAD` is portable, but I'm not sure about older versions (it is okay in 1.8 in CentOS 7 at least.)
@fiesh / @WSoptics What was the original reason not to use `git clone --depth=xxx --branch something …` directly? From looking at old manpages it seems that is supported since git 1.7 (branches) or git 1.8 (tags).

Looking more at this change i think this also break having revision point at an commit id, because these are not fetchable directly but only branches and tags.

So my suggested fix would be to switch back to the previous code but special case wraps using `depth` to use `git clone --depth=xxx --branch something …` and skip the `git checkout …` step.
@textshell the reason is that `git clone --depth` only works for branches, not for commits. 
According to the documentation `git clone --depth` works with branches and tags. With tags it also only sets `FETCH_HEAD` on my system (git 2.11.0). Testing with commit ids yields an error¹ as expected.

Do you have some examples where your approach works but `git clone --depth` does not? Somewhat related, we really need tests for this, because it seems git has it's share on non obvious edge cases.

¹
```
$ git fetch origin 0008b326ffc39c8
fatal: Couldn't find remote ref 0008b326ffc39c8
```
@textshell Like I said, `clone` does not work for commits ids to the best of my knowledge. 
@fiesh At least in the default config, clone does not work for commit ids. But as far as i know `git fetch` doesn't work either.

The referenced stack overflow post says an special option is needed `uploadpack.allowReachableSHA1InWant=true`, but even with this option i don't get it to work when using a local (file:// repo) not does this work in my testing with github.com, gitlab.com nor another gitlab instance. So i can't really test if clone does not work the same with that option.

If you disagree please provide an example where `git fetch` works but `git clone` does not. Preferably on a publicly accessible repository or with clear instructions to reproduce.
@textshell I set the option for our local Gitlab installation and it worked on the spot, that's what I tested the new wrap option with.  I don't know of any public repository supporting this, possibly it comes at a computational cost, which might also be why it's not activated by default. 

So I disagree, but of course you have to enable the git option for your own installation.

Note that the necessity for the git server option is also documented in Meson's documentation. 
I found the missing bit. Only *full* commit id·s work. But then i can reproduce git fetch working but not git clone.

I think unconditionally using the fetch based procedure is bad, because it seems to break centos 7 and it definitivly breaks git describe and workflows that expect tags and branches to be available.

So i think meson should either special case `depth` set or even only `depth` and a full commit id to use the fetch based code path.

Also it would be nice to add something to the documentation that full commit id·s is what is supported by this option.

@fiesh do you see any problems going that way apart from the added complexity?
@textshell can you please explain how the `fetch` approach differs for git 2? For if it doesn't, maybe the commands should better depend on the git version being used. 

Also if git 1.8 is supposed to be supported, I guess there should be a CI pipeline for it?
Yes, it would be great to have a lot more testing for this.

The fetch workflow does not populate other branches and tags. At least tags are commonly needed for various `git describe` like workflows that depend on the tags being there. The most important here will be meson's own vcs_tag command. But there are likely very many projects using custom scripts also.

Resolving patch:
diff --git a/mesonbuild/wrap/wrap.py b/mesonbuild/wrap/wrap.py
--- a/mesonbuild/wrap/wrap.py
+++ b/mesonbuild/wrap/wrap.py
@@ -226,27 +226,56 @@ def get_file(self):
             self.apply_patch()
 
     def get_git(self):
-        # git doesn't support directly cloning shallowly for commits,
-        # so we follow https://stackoverflow.com/a/43136160
-        subprocess.check_call(['git', 'init', self.directory], cwd=self.subdir_root)
-        subprocess.check_call(['git', 'remote', 'add', 'origin', self.wrap.get('url')],
-                              cwd=self.dirname)
-        git_options = []
         revno = self.wrap.get('revision')
-        if self.wrap.values.get('depth', '') != '':
-            git_options.extend(['--depth', self.wrap.values.get('depth')])
-        subprocess.check_call(['git', 'fetch'] + git_options + ['origin', revno],
-                              cwd=self.dirname)
-        subprocess.check_call(['git', 'checkout', revno], cwd=self.dirname)
-        if self.wrap.values.get('clone-recursive', '').lower() == 'true':
-            subprocess.check_call(['git', 'submodule', 'update',
-                                   '--init', '--checkout', '--recursive'] + git_options,
+        is_shallow = self.wrap.values.get('depth', '') != ''
+        # for some reason git only allows commit ids to be shallowly fetched by fetch not with clone
+        if is_shallow and self.is_git_full_commit_id(revno):
+            # git doesn't support directly cloning shallowly for commits,
+            # so we follow https://stackoverflow.com/a/43136160
+            subprocess.check_call(['git', 'init', self.directory], cwd=self.subdir_root)
+            subprocess.check_call(['git', 'remote', 'add', 'origin', self.wrap.get('url')],
                                   cwd=self.dirname)
-        push_url = self.wrap.values.get('push-url')
-        if push_url:
-            subprocess.check_call(['git', 'remote', 'set-url',
-                                   '--push', 'origin', push_url],
+            revno = self.wrap.get('revision')
+            subprocess.check_call(['git', 'fetch', '--depth', self.wrap.values.get('depth'), 'origin', revno],
                                   cwd=self.dirname)
+            subprocess.check_call(['git', 'checkout', revno], cwd=self.dirname)
+            if self.wrap.values.get('clone-recursive', '').lower() == 'true':
+                subprocess.check_call(['git', 'submodule', 'update',
+                                       '--init', '--checkout', '--recursive', '--depth', self.wrap.values.get('depth')],
+                                      cwd=self.dirname)
+            push_url = self.wrap.values.get('push-url')
+            if push_url:
+                subprocess.check_call(['git', 'remote', 'set-url',
+                                       '--push', 'origin', push_url],
+                                      cwd=self.dirname)
+        else:
+            if not is_shallow:
+                subprocess.check_call(['git', 'clone', self.wrap.get('url'),
+                                       self.directory], cwd=self.subdir_root)
+                if revno.lower() != 'head':
+                    if subprocess.call(['git', 'checkout', revno], cwd=self.dirname) != 0:
+                        subprocess.check_call(['git', 'fetch', self.wrap.get('url'), revno], cwd=self.dirname)
+                        subprocess.check_call(['git', 'checkout', revno], cwd=self.dirname)
+            else:
+                subprocess.check_call(['git', 'clone', '--depth', self.wrap.values.get('depth'),
+                                       '--branch', revno,
+                                       self.wrap.get('url'),
+                                       self.directory], cwd=self.subdir_root)
+            if self.wrap.values.get('clone-recursive', '').lower() == 'true':
+                subprocess.check_call(['git', 'submodule', 'update',
+                                       '--init', '--checkout', '--recursive', '--depth', self.wrap.values.get('depth')],
+                                      cwd=self.dirname)
+            push_url = self.wrap.values.get('push-url')
+            if push_url:
+                subprocess.check_call(['git', 'remote', 'set-url',
+                                       '--push', 'origin', push_url],
+                                      cwd=self.dirname)
+
+    def is_git_full_commit_id(self, revno):
+        result = False
+        if len(revno) in (40, 64): # 40 for sha1, 64 for upcoming sha256
+            result = all((ch in '0123456789AaBbCcDdEeFf' for ch in revno))
+        return result
 
     def get_hg(self):
         revno = self.wrap.get('revision')


Test patch: