Repository: conda/conda-build

Issue:
Leaving Source/git_rev key blank results in a spurious git revision being cloned
I left the Source/git_rev: field blank when using a local git repo, and was surprised to see that conda-build cloned a seemingly random commit (it was even on a different branch than the one I was working on).

When I removed the git_rev line altogether, or replaced it with HEAD, then I got the expected behavior (cloning HEAD from the current branch).

I think the code responsible for this behavior may be here:
https://github.com/conda/conda-build/blob/7d407c5f80863de4d07084ee14f7c7ee6bd064ea/conda_build/source.py#L294

We're not getting the default of 'HEAD' on an empty key, possibly because of: 
https://github.com/conda/conda-build/blob/7d407c5f80863de4d07084ee14f7c7ee6bd064ea/conda_build/metadata.py#L327-L329

If leaving a key empty like this is considered bad practice, then perhaps an error would be a good fix? I would also like it if I could see the actual git commands being run. Is there a debug flag that would let me see this?

See command output below showing the difference in behavior between the empty key and the missing key. I don't _always_ get a bad commit in the empty key case, but I do always see "Checkout: None", which suggests that the problem is not intermittent, it just depends somehow on the state of my git repo. 

___
### Success: Source/git_rev key missing
Running with no Source/git_rev key in meta.yaml. Note line reading "Checkout: 'HEAD', and matching commit 7571c28 at command line and in conda-build run:
```
$ head -12  conda-recipe/meta.yaml
#Automatically generating npm conda package
{% set data = load_npm() %}
package:
  name: {{data.get('name').lower()}}
  version: {{data.get('version')}}

source:
  git_url: .. # relative to recipe dir

build:
  number: {% block build_number %}1{% endblock %}
  script:

$ git status
On branch conda-activate
Your branch is up-to-date with 'origin/conda-activate'.
nothing to commit, working tree clean

$ git log -n1
commit 7571c2807494826167b8d9d8115ce1dec9b42863
Author: Tom Davidson <tjd@alum.mit.edu>
Date:   Sat Jun 30 01:01:45 2018 -0700

    conda-activate: Much simpler approach, don't save any env vars. Overly verbose messages

$ conda-build conda-recipe/
WARNING: The conda.compat module is deprecated and will be removed in a future release.
WARNING: The conda.install module is deprecated and will be removed in a future release.
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Attempting to finalize metadata for mountainlab-js
INFO:conda_build.metadata:Attempting to finalize metadata for mountainlab-js
Solving environment: ...working... done
Solving environment: ...working... done
BUILD START: ['mountainlab-js-0.4.1-1.tar.bz2']
Solving environment: ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /Users/tjd/anaconda3/conda-bld/mountainlab-js_1530346291912/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol


The following NEW packages will be INSTALLED:

    libcxx:    4.0.1-h579ed51_0
    libcxxabi: 4.0.1-hebd6815_0
    nodejs:    8.11.1-h0a44026_0

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
From /Users/tjd/Src/mountainlab-js
 * [new ref]                    -> _conda_cache_origin_head
Cloning into '/Users/tjd/anaconda3/conda-bld/mountainlab-js_1530346291912/work'...
done.
checkout: 'HEAD'
Your branch is up-to-date with 'origin/_conda_cache_origin_head'.
==> git log -n1 <==

fatal: No names found, cannot describe anything.
commit 7571c2807494826167b8d9d8115ce1dec9b42863
Author: Tom Davidson <tjd@alum.mit.edu>
Date:   Sat Jun 30 01:01:45 2018 -0700

    conda-activate: Much simpler approach, don't save any env vars. Overly verbose messages

==> git describe --tags --dirty <==

commit 7571c2807494826167b8d9d8115ce1dec9b42863
Author: Tom Davidson <tjd@alum.mit.edu>
Date:   Sat Jun 30 01:01:45 2018 -0700

    conda-activate: Much simpler approach, don't save any env vars. Overly verbose messages

==> git status <==

On branch _conda_cache_origin_head
Your branch is up-to-date with 'origin/_conda_cache_origin_head'.
nothing to commit, working tree clean

[...]
```

___

### Failure: Source/git_rev key blank
Note line reading 'Checkout: None', and mismatch between branch and commit between command line [conda activate 7571c28] and the conda-build run [conda-build-tgz b190e6c]

```
$ head -12 conda-recipe/meta.yaml
#Automatically generating npm conda package
{% set data = load_npm() %}
package:
  name: {{data.get('name').lower()}}
  version: {{data.get('version')}}

source:
  git_url: .. # relative to recipe dir
  git_rev:

build:
  number: {% block build_number %}1{% endblock %}

$ git status
On branch conda-activate
Your branch is up-to-date with 'origin/conda-activate'.
nothing to commit, working tree clean

(base) eena:mountainlab-js tjd$ conda-build conda-recipe/
WARNING: The conda.compat module is deprecated and will be removed in a future release.
WARNING: The conda.install module is deprecated and will be removed in a future release.
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Attempting to finalize metadata for mountainlab-js
INFO:conda_build.metadata:Attempting to finalize metadata for mountainlab-js
Solving environment: ...working... done
Solving environment: ...working... done
BUILD START: ['mountainlab-js-0.4.1-1.tar.bz2']
Solving environment: ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /Users/tjd/anaconda3/conda-bld/mountainlab-js_1530345875480/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol


The following NEW packages will be INSTALLED:

    libcxx:    4.0.1-h579ed51_0
    libcxxabi: 4.0.1-hebd6815_0
    nodejs:    8.11.1-h0a44026_0

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
remote: Counting objects: 14, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 14 (delta 4), reused 0 (delta 0)
Unpacking objects: 100% (14/14), done.
From /Users/tjd/Src/mountainlab-js
   e9e472c..7571c28  conda-activate        -> conda-activate
   e9e472c..7571c28  origin/conda-activate -> origin/conda-activate
Cloning into '/Users/tjd/anaconda3/conda-bld/mountainlab-js_1530345875480/work'...
done.
checkout: None
==> git log -n1 <==

fatal: No names found, cannot describe anything.
commit b190e6cfbeab6fc4a693f2a17678c08c38b57b11
Author: Tom Davidson <tjd@alum.mit.edu>
Date:   Thu Jun 28 10:26:35 2018 -0700

    fixup! Add recipe to build a conda package from local git repo

==> git describe --tags --dirty <==

commit b190e6cfbeab6fc4a693f2a17678c08c38b57b11
Author: Tom Davidson <tjd@alum.mit.edu>
Date:   Thu Jun 28 10:26:35 2018 -0700

    fixup! Add recipe to build a conda package from local git repo

==> git status <==

On branch conda-build-tgz
Your branch is up-to-date with 'origin/conda-build-tgz'.
nothing to commit, working tree clean

source tree in: /Users/tjd/anaconda3/conda-bld/mountainlab-js_1530345875480/work
[...]
```

___

### More info:
```
$ conda-build -V
WARNING: The conda.compat module is deprecated and will be removed in a future release.
WARNING: The conda.install module is deprecated and will be removed in a future release.
conda-build 3.10.5

$ conda -V
conda 4.6.0a2
```


Discussion:
(FWIW, I think this is related to an old issue: #803.)  My guess is that the "seemingly random" commit was whatever the cache in `.../conda-bld/git_cache/.../mountainlab-js` happened to be left with (that is, whatever commit you were using when you previously built your recipe).

Anyway, I think you're right about the line to change.  Maybe the following patch would suffice?

```patch
-git_ref = source_dict.get('git_rev', 'HEAD')
+git_ref = source_dict.get('git_rev', 'HEAD') or 'HEAD'
```

Alternatively, (or additionally), perhaps a similar line is appropriate at the top of [`git_mirror_checkout_recursive`](https://github.com/conda/conda-build/blob/7d407c5f80863de4d07084ee14f7c7ee6bd064ea/conda_build/source.py#L165).  Your example shows that `git_ref=None` is not a valid input to that function.  It should start with:

```python
git_ref = git_ref or 'HEAD'
```


Your explanation about the cache explains the intermittent behavior I was seeing. I can now trigger this behavior reliably by checking out a new branch and running conda-build.
```
git rev-parse --abbrev-ref HEAD # master
git checkout -b buildtest1
git commit --allow-empty -m "buildtest1 commit"
conda-build conda-recipe # builds from master 
```

(The Checkout value of 'None' seems to be caused by `git rev-parse` returning the empty string when run with no argument; and then the test for git_ref != 'HEAD' causes all the   _conda_cache_origin_head logic to be skipped).

I have tested that both of your proposed fixes seem to work. The fix in `git_mirror_checkout_recursive` seems the most robust. 

Thanks!
Nice.  Will you make a PR, then?

(Also: It's not obvious to me how to add a test case for this, since it requires manipulating the `git_cache` before running the test. Maybe it's not worth the trouble.  A conda-build developer can comment on the PR if they want one.)
Sure, I can make a PR.

> On Jul 4, 2018, at 10:43 AM, Stuart Berg <notifications@github.com> wrote:
> 
> Nice. Will you make a PR, then?
> 
> (Also: It's not obvious to me how to add a test case for this, since it requires manipulating the git_cache before running the test. Maybe it's not worth the trouble. A conda-build developer can comment on the PR if they want one.)
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub <https://github.com/conda/conda-build/issues/2992#issuecomment-402533153>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAK3c-HfxWAlIXl9Lkfyvf-50H9pGgJMks5uDP6mgaJpZM4U955O>.
> 



Resolving patch:
diff --git a/conda_build/source.py b/conda_build/source.py
--- a/conda_build/source.py
+++ b/conda_build/source.py
@@ -195,6 +195,9 @@ def git_mirror_checkout_recursive(git, mirror_dir, checkout_dir, git_url, git_ca
     git_mirror_dir = convert_path_for_cygwin_or_msys2(git, mirror_dir).rstrip('/')
     git_checkout_dir = convert_path_for_cygwin_or_msys2(git, checkout_dir).rstrip('/')
 
+    # Set default here to catch empty dicts
+    git_ref = git_ref or 'HEAD'
+
     mirror_dir = mirror_dir.rstrip('/')
     if not isdir(os.path.dirname(mirror_dir)):
         os.makedirs(os.path.dirname(mirror_dir))
@@ -291,7 +294,7 @@ def git_source(source_dict, git_cache, src_dir, recipe_path=None, verbose=True):
 
     git_url = source_dict['git_url']
     git_depth = int(source_dict.get('git_depth', -1))
-    git_ref = source_dict.get('git_rev', 'HEAD')
+    git_ref = source_dict.get('git_rev') or 'HEAD'
 
     if git_url.startswith('.'):
         # It's a relative path from the conda recipe


Test patch: