Repository: conda/conda-build

Issue:
Version 2.1 breaks git environment variables
The following recipe:
https://github.com/NNPDF/reportengine/tree/master/conda-recipe

works with conda-build 2.0 when I type:

```
conda build conda-recipe
```

from the root folder of the project.

Instead with 2.1 the environment variables `GIT_DESCRIBE_TAG` and `GIT_DESCRIBE_NUMBER` are undefined and what I see is:

```
reportengine (master *%)$ conda build conda-recipe
Cloning into bare repository '/home/zah/anaconda3/conda-bld/git_cache/home/zah/phd/reportengine'...
done.
Cloning into '/home/zah/anaconda3/conda-bld/conda-recipe_1484075474810/work'...
done.
checkout: 'HEAD'
Your branch is up-to-date with 'origin/master'.
==> git log -n1 <==

commit bb51ceb565198c1859d0bdd38fc8f753d089c508
Author: Zahari <zaharid@gmail.com>
Date:   Mon Jan 2 15:16:02 2017 +0100

    Refactor templateparser
    
    Move parsing to individual function.


==> git describe --tags --dirty <==

v0.9-14-gbb51ceb


==> git status <==

On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean


Traceback (most recent call last):
  File "/home/zah/anaconda3/bin/conda-build", line 6, in <module>
    sys.exit(conda_build.cli.main_build.main())
  File "/home/zah/anaconda3/lib/python3.5/site-packages/conda_build/cli/main_build.py", line 312, in main
    execute(sys.argv[1:])
  File "/home/zah/anaconda3/lib/python3.5/site-packages/conda_build/cli/main_build.py", line 304, in execute
    already_built=None, config=config, noverify=args.no_verify)
  File "/home/zah/anaconda3/lib/python3.5/site-packages/conda_build/api.py", line 86, in build
    need_source_download=need_source_download, config=config)
  File "/home/zah/anaconda3/lib/python3.5/site-packages/conda_build/build.py", line 1413, in build_tree
    config=config)
  File "/home/zah/anaconda3/lib/python3.5/site-packages/conda_build/render.py", line 157, in render_recipe
    config=config)
  File "/home/zah/anaconda3/lib/python3.5/site-packages/conda_build/render.py", line 90, in parse_or_try_download
    metadata.parse_again(config=config, permit_undefined_jinja=False)
  File "/home/zah/anaconda3/lib/python3.5/site-packages/conda_build/metadata.py", line 431, in parse_again
    self.meta = parse(self._get_contents(permit_undefined_jinja, config=config),
  File "/home/zah/anaconda3/lib/python3.5/site-packages/conda_build/metadata.py", line 770, in _get_contents
    permit_undefined_jinja=permit_undefined_jinja))
  File "/home/zah/anaconda3/lib/python3.5/site-packages/conda_build/jinja_context.py", line 213, in context_processor
    ctx = get_environ(config=config, m=initial_metadata)
  File "/home/zah/anaconda3/lib/python3.5/site-packages/conda_build/environ.py", line 222, in get_dict
    d.update(meta_vars(m, config))
  File "/home/zah/anaconda3/lib/python3.5/site-packages/conda_build/environ.py", line 331, in meta_vars
    d['PKG_VERSION'] = meta.version()
  File "/home/zah/anaconda3/lib/python3.5/site-packages/conda_build/metadata.py", line 547, in version
    assert not res.startswith('.'), "Version can't start with leading period -  got %s" % res
AssertionError: Version can't start with leading period -  got .0
```


Discussion:
could you please post examples of what is broken, and what error messages look like?  Our tests use git environment variables extensively in Jinja2 templates.
Sorry I submitted the issue to early, by error. The message should contain the error now.
Ah, thanks.  This is a duplicate of #1640 and is addressed by #1644.  Closing this issue.  New release coming later today.
This is still broken for me in 2.1.1.
Thanks. I'll test again with your recipe and fix.
@Zaharid the tricky thing here is that when you use ``environ.get`` with Jinja2 variables, then conda-build is unable to determine when these variables are actually undefined.  If you change your recipe to be 

```
  version: "{{ GIT_DESCRIBE_TAG[1:] }}.{{ GIT_DESCRIBE_NUMBER }}"
```

this works - because the Jinja2 variables are known to be undefined.  I don't think there's a way for me to fix your old way of doing things, unfortunately.  It's just that the old version of conda-build happily allows the invalid version, while the new version ultimately disallows it, but temporarily permits it until there is a restrictive Jinja2 pass that does not allow undefined variables.  As far as Jinja2 is concerned, any variable obtained with environ.get, rather than direct access, is always defined.

I don't see a way forward on this without changing your recipe, unfortunately.
Couldn't environ be patched to do the same thing as the jinja variables?  Maybe subclassing UserDict like:

```python
import os
from collections import UserDict

def check(item):
    print(item)


variables = {'GIT_DESCRIBE_TAG', 'GIT_DESCRIBE_NUMBER'}
class JinjaEnviron(UserDict):
    def __getitem__(self, item):
        if item in variables:
            check(item)
        super().__getitem__(item)
        
environ = JinjaEnviron(os.environ)

environ.get('GIT_DESCRIBE_TAG')
```

Using `environ` used to be de documented way of doing this, and it is still suggested in some cases here:
http://conda.pydata.org/docs/building/meta-yaml.html#templating-with-jinja
I could update my recipes, and would be nice to do if I get better error messages for undefined variables (for example it is quite annoying that Travis pulls by default only the 50 latest commits which could miss the latest tag and then one ends up with a bogus version), but seems to me this is bound to cause other problems...

Also the reason why there is this error message in the first place is #1586 (is there any other reason in newer conda build versions?). Maybe there is a more targeted check to avoid building recipes infinitely?
patching environ to be a different dictionary totally defeats the purpose of having it as a dictionary in the first place, as far as I can tell.  The point is to be able to set a default value.  I really question the value of setting default values at all, but if someone wants to do it, who am I to stop them?

You're right that #1586 is the reason for this check.  I think that our discussion in that issue still stands: conda-build really, really needs DAG-based building.  That is something I'll work on as soon as I can, but there are some paid projects that I need to finish first ( #1585 ).
Defaults still work if the check function above is made to raise a KeyError on failure.
```python
import os
from collections import UserDict

class CondaJinjaError(KeyError): pass

def check(item):
    print("Too bad")
    raise CondaJinjaError(item) 


variables = {'GIT_DESCRIBE_TAG', 'GIT_DESCRIBE_NUMBER'}
class JinjaEnviron(UserDict):
    def __getitem__(self, item):
        if item in variables:
            check(item)
        super().__getitem__(item)
        
environ = JinjaEnviron(os.environ)

assert environ.get('GIT_DESCRIBE_TAG', 'default') == 'default'

#Raises
environ['GIT_DESCRIBE_TAG']
```

I agree defaults are probably not very useful. I wrote them once to debug some problem with the CI, but later copied that stub in many places...
Sure, your example is correct, but because you have the default there, the Jinja2 code still won't raise that error, and Jinja2 still won't see the variable as undefined.  Where there's a default, it can't ever be undefined. =(

Resolving patch:
diff --git a/conda_build/build.py b/conda_build/build.py
--- a/conda_build/build.py
+++ b/conda_build/build.py
@@ -1046,12 +1046,14 @@ def build(m, config, post=None, need_source_download=True, need_reparse_in_env=F
                                                                 no_download_source=False,
                                                                 force_download=True,
                                                                 config=config)
+            m.final = True
             assert not need_source_download, "Source download failed.  Please investigate."
             if m.uses_jinja:
                 print("BUILD START (revised):", m.dist())
 
         if need_reparse_in_env:
             reparse(m, config=config)
+            m.final = True
             print("BUILD START (revised):", m.dist())
 
         print("Package:", m.dist())
diff --git a/conda_build/metadata.py b/conda_build/metadata.py
--- a/conda_build/metadata.py
+++ b/conda_build/metadata.py
@@ -437,6 +437,16 @@ def __init__(self, path, config=None):
         self.parse_again(config=config, permit_undefined_jinja=True)
         self.config.disable_pip = self.disable_pip
 
+    @property
+    def final(self):
+        return self.get_value('extra/final')
+
+    @final.setter
+    def final(self, boolean):
+        extra = self.meta.get('extra', {})
+        extra['final'] = boolean
+        self.meta['extra'] = extra
+
     @property
     def disable_pip(self):
         return 'build' in self.meta and 'disable_pip' in self.meta['build']
@@ -582,8 +592,8 @@ def version(self):
         if res is None:
             sys.exit("Error: package/version missing in: %r" % self.meta_path)
         check_bad_chrs(res, 'package/version')
-        assert self.undefined_jinja_vars or not res.startswith('.'), "Fully-rendered version can't\
-        start with leading period -  got %s" % res
+        if self.final and res.startswith('.'):
+            raise ValueError("Fully-rendered version can't start with period -  got %s", res)
         return res
 
     def build_number(self):
diff --git a/conda_build/render.py b/conda_build/render.py
--- a/conda_build/render.py
+++ b/conda_build/render.py
@@ -164,6 +164,8 @@ def render_recipe(recipe_path, config, no_download_source=False):
 
     if need_cleanup:
         rm_rf(recipe_dir)
+    if not need_download or need_reparse_in_env:
+        m.final = True
 
     return m, need_download, need_reparse_in_env
 


Test patch:
diff --git a/tests/test-recipes/metadata/_source_git_jinja2_relative_git_url/meta.yaml b/tests/test-recipes/metadata/_source_git_jinja2_relative_git_url/meta.yaml
--- a/tests/test-recipes/metadata/_source_git_jinja2_relative_git_url/meta.yaml
+++ b/tests/test-recipes/metadata/_source_git_jinja2_relative_git_url/meta.yaml
@@ -1,6 +1,6 @@
 package:
   name: flex
-  version: {{ environ.get('GIT_DESCRIBE_TAG', '0.0') }}
+  version: {{ environ.get('GIT_DESCRIBE_TAG', '.0') }}
 
 build:
   number: {{ environ.get('GIT_DESCRIBE_NUMBER', 0) }}
diff --git a/tests/test-recipes/metadata/extra_freeform_metadata/run_test.py b/tests/test-recipes/metadata/extra_freeform_metadata/run_test.py
--- a/tests/test-recipes/metadata/extra_freeform_metadata/run_test.py
+++ b/tests/test-recipes/metadata/extra_freeform_metadata/run_test.py
@@ -16,7 +16,8 @@ def main():
         source = yaml.load(fh)
 
     assert source['extra'] == {"custom": "metadata",
-                               "however": {"we": "want"}}
+                               "however": {"we": "want"},
+                               "final": True}
 
 
 if __name__ == '__main__':
diff --git a/tests/test_metadata.py b/tests/test_metadata.py
--- a/tests/test_metadata.py
+++ b/tests/test_metadata.py
@@ -112,7 +112,8 @@ def test_numpy(self):
 
 def test_disallow_leading_period_in_version(test_metadata):
     test_metadata.meta['package']['version'] = '.ste.ve'
-    with pytest.raises(AssertionError):
+    test_metadata.final = True
+    with pytest.raises(ValueError):
         test_metadata.version()