Repository: conda/conda-build

Issue:
turn off pyc compilation?
Is there a way to do this? why?! you may ask?, well

we build conda packages to deploy jobs
we build fat packages (meaning, they contain lots of stuff, which might not be used for one thing or the other)
So the pyc compilation process actually adds alot of overhead to deploying jobs, also, pacakge builds break if we have invalid python in the package

thanks



Discussion:
From what I can tell, there's no way to do this right now.  Should be easy enough to add a [field](https://github.com/conda/conda-build/blob/master/conda_build/metadata.py#L254) to `meta.yaml`, and then conditionally skip the [`compile_missing_pyc`](https://github.com/conda/conda-build/blob/master/conda_build/post.py#L151) step.

I think this would be an easy PR to review, but don't forget to update the docs, too.

ok thanks I'll take care of this

Maybe instead of adding some new feature to `meta.yaml` the existing python environment variable [`PYTHONDONTWRITEBYTECODE`](https://docs.python.org/2/using/cmdline.html#envvar-PYTHONDONTWRITEBYTECODE) can be used.

It turns out there is more to the story as conda is doing its own additional compilation internally, which seems a bit counterintuitive IMHO. I wonder if it is worthwhile to just permanently remove this internal compilation step. It seems to cause problems with complex packages like [Django](https://github.com/conda/conda-build/issues/709) and Pylint (and friends) that template Python files. This will allow users to actually control compilation in their recipes the way Python intended.

@jakirkham I'm +1 on removing the compilation of pyc files entirely.  There is no need to distribute them in the package.  They can be compiled just as easily at install time (because write permissions are guaranteed).

> They can be compiled just as easily at install time

True. I know RPMs do pyc compilation at build time, and they even hardlink pyc and pyo files when they are the same. The advantage of doing the compilation at build time is that environments can share (through hard links) the pyc files, whereas if they are compiled at install time, that adds an extra complexity to the currently trivial linking step.

Do you have an example of this @pelson?

Also, should add that this variable `sys.dont_write_bytecode` does the same as the aforementioned environment variable.

On a more general note, `conda` has grown to support many languages for building and install (I saw Lua support being added recently). Personally, I'd like to avoid cluttering `meta.yaml` with more language specific stuff than necessary. I would prefer to follow the "one right way to do things mantra" and use  Python's already supported mechanism for this behavior (incorporating it into our logic, if needed) or simply disable `.pyc` generation given the various problems it has caused.

I agree that we can probably avoid adding anything additional to `meta.yaml`.  I'd think that `PYTHONDONTWRITEBYTECODE` should suffice.  At least that would be my hope.

Can someone with more history here, maybe @ilanschnell or @stuarteberg, talk to the purpose of [`compile_missing_pyc`](https://github.com/conda/conda-build/blob/master/conda_build/post.py#L130)? It's undocumented in the code, and it's not obvious to me why it exists.

`pyc` files exist for a reason, and in production environments, it's likely that they won't be able to be written by the running process.  They either need to be built at build time or at install time.  For conda packages with a python version attached, they should be built _appropriately_ at build time.  For noarch python conda packages, they need to be built at install time. (Correct me if I'm wrong, but I think py3 bytecode has differences from py2 bytecode.)

That function has been there for as long as this code has been public AFAICT. Here it is in the second commit of the repo, first commit with code ( https://github.com/conda/conda-build/commit/fe7f773010f4c8c200298c83a4164ca404626d52#diff-60400bbbdb51124a0a8a06ca17ecec5cR106 ).

Also, you are correct on pyc files. Though they can differ between minor Python versions too. There is good background on how this work in this PEP ( https://www.python.org/dev/peps/pep-3147 ).

`compile_missing_pyc` has actually even been around longer.  The first public version of conda-build was part of conda itself, and was added in June 2013 in conda version 1.7.  That first conda-build had `compile_missing_pyc` already.  It wasn't till January 2014 that conda-build became its own project.

If we replace [this call](https://github.com/conda/conda-build/blame/4fa519d4c448c2edd81d7355e859c9e25755d35a/conda_build/utils.py#L56) with `Popen`, we can at least pass an environment variable to disable compilation optionally, which would solve the original issue.

Any thoughts on this one? Would be really nice to come to a solution here.


Resolving patch:
diff --git a/conda_build/build.py b/conda_build/build.py
--- a/conda_build/build.py
+++ b/conda_build/build.py
@@ -766,7 +766,8 @@ def build(m, config, post=None, need_source_download=True, need_reparse_in_env=F
                      prefix=config.build_prefix,
                      config=config,
                      preserve_egg_dir=bool(m.get_value('build/preserve_egg_dir')),
-                     noarch=m.get_value('build/noarch'))
+                     noarch=m.get_value('build/noarch'),
+                     skip_compile_pyc=m.get_value('build/skip_compile_pyc'))
 
         # The post processing may have deleted some files (like easy-install.pth)
         files2 = prefix_files(prefix=config.build_prefix)
diff --git a/conda_build/metadata.py b/conda_build/metadata.py
--- a/conda_build/metadata.py
+++ b/conda_build/metadata.py
@@ -237,6 +237,7 @@ def parse(data, config, path=None):
     'build/noarch_python': bool,
     'build/detect_binary_files_with_prefix': bool,
     'build/skip': bool,
+    'build/skip_compile_pyc': list,
     'app/own_environment': bool
 }
 
@@ -305,8 +306,8 @@ def _git_clean(source_meta):
               'features', 'track_features', 'preserve_egg_dir',
               'no_link', 'binary_relocation', 'script', 'noarch', 'noarch_python',
               'has_prefix_files', 'binary_has_prefix_files', 'ignore_prefix_files',
-              'detect_binary_files_with_prefix', 'rpaths', 'script_env',
-              'always_include_files', 'skip', 'msvc_compiler',
+              'detect_binary_files_with_prefix', 'skip_compile_pyc', 'rpaths',
+              'script_env', 'always_include_files', 'skip', 'msvc_compiler',
               'pin_depends', 'include_recipe'  # pin_depends is experimental still
               ],
     'requirements': ['build', 'run', 'conflicts'],
diff --git a/conda_build/post.py b/conda_build/post.py
--- a/conda_build/post.py
+++ b/conda_build/post.py
@@ -8,6 +8,7 @@
 import mmap
 import re
 import os
+import fnmatch
 from os.path import (basename, dirname, join, splitext, isdir, isfile, exists,
                      islink, realpath, relpath, normpath)
 import stat
@@ -167,9 +168,14 @@ def rm_pyc(files, prefix):
             os.unlink(os.path.join(prefix, fn))
 
 
-def compile_missing_pyc(files, cwd, python_exe):
+def compile_missing_pyc(files, cwd, python_exe, skip_compile_pyc=()):
     compile_files = []
-    for fn in files:
+    skip_compile_pyc_n = [os.path.normpath(skip) for skip in skip_compile_pyc]
+    skipped_files = set()
+    for skip in skip_compile_pyc_n:
+        skipped_files.update(set(fnmatch.filter(files, skip)))
+    unskipped_files = set(files) - skipped_files
+    for fn in unskipped_files:
         # omit files in Library/bin, Scripts, and the root prefix - they are not generally imported
         if sys.platform == 'win32':
             if any([fn.lower().startswith(start) for start in ['library/bin', 'library\\bin',
@@ -183,18 +189,20 @@ def compile_missing_pyc(files, cwd, python_exe):
                 os.path.dirname(fn) + cache_prefix + os.path.basename(fn) + 'c' not in files):
             compile_files.append(fn)
 
-    if compile_files and os.path.isfile(python_exe):
-        print('compiling .pyc files...')
-        for f in compile_files:
-            call([python_exe, '-Wi', '-m', 'py_compile', f], cwd=cwd)
-
+    if compile_files:
+        if not os.path.isfile(python_exe):
+            print('compiling .pyc files... failed as no python interpreter was found')
+        else:
+            print('compiling .pyc files...')
+            for f in compile_files:
+                call([python_exe, '-Wi', '-m', 'py_compile', f], cwd=cwd)
 
-def post_process(files, prefix, config, preserve_egg_dir=False, noarch=False):
+def post_process(files, prefix, config, preserve_egg_dir=False, noarch=False, skip_compile_pyc=()):
     rm_pyo(files, prefix)
     if noarch:
         rm_pyc(files, prefix)
     else:
-        compile_missing_pyc(files, cwd=prefix, python_exe=config.build_python)
+        compile_missing_pyc(files, cwd=prefix, python_exe=config.build_python, skip_compile_pyc=skip_compile_pyc)
     remove_easy_install_pth(files, prefix, config, preserve_egg_dir=preserve_egg_dir)
     rm_py_along_so(prefix)
 


Test patch:
diff --git a/tests/test-recipes/metadata/skip_compile_pyc/README b/tests/test-recipes/metadata/skip_compile_pyc/README
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/skip_compile_pyc/README
@@ -0,0 +1 @@
+Simple package to test skip_compile_pyc package building.
diff --git a/tests/test-recipes/metadata/skip_compile_pyc/compile_pyc.py b/tests/test-recipes/metadata/skip_compile_pyc/compile_pyc.py
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/skip_compile_pyc/compile_pyc.py
@@ -0,0 +1,2 @@
+import os
+
diff --git a/tests/test-recipes/metadata/skip_compile_pyc/meta.yaml b/tests/test-recipes/metadata/skip_compile_pyc/meta.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/skip_compile_pyc/meta.yaml
@@ -0,0 +1,20 @@
+package:
+  name: skip_compile_pyc
+  version: "1.0"
+
+source:
+  path: .
+
+requirements:
+  build:
+    - python
+
+build:
+  script:
+    - cp -rf "${SRC_DIR}"/* "${PREFIX}"/  # [unix]
+    - xcopy /s %SRC_DIR% %PREFIX%         # [win]
+  skip_compile_pyc:
+    # rec_glob is used to find files:
+    - sub/skip*
+    # test that path normalization happens:
+    - ./sub/../skip_compile_pyc.py
diff --git a/tests/test-recipes/metadata/skip_compile_pyc/skip_compile_pyc.py b/tests/test-recipes/metadata/skip_compile_pyc/skip_compile_pyc.py
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/skip_compile_pyc/skip_compile_pyc.py
@@ -0,0 +1,2 @@
+import os
+
diff --git a/tests/test-recipes/metadata/skip_compile_pyc/sub/compile_pyc.py b/tests/test-recipes/metadata/skip_compile_pyc/sub/compile_pyc.py
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/skip_compile_pyc/sub/compile_pyc.py
@@ -0,0 +1,2 @@
+import os
+
diff --git a/tests/test-recipes/metadata/skip_compile_pyc/sub/skip_compile_pyc.py b/tests/test-recipes/metadata/skip_compile_pyc/sub/skip_compile_pyc.py
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/skip_compile_pyc/sub/skip_compile_pyc.py
@@ -0,0 +1,2 @@
+import os
+
diff --git a/tests/test_api_build.py b/tests/test_api_build.py
--- a/tests/test_api_build.py
+++ b/tests/test_api_build.py
@@ -16,6 +16,7 @@
 from binstar_client.errors import NotFound
 import pytest
 import yaml
+import tarfile
 
 from conda_build import api, exceptions
 from conda_build.utils import (copy_into, on_win, check_call_env, convert_path_for_cygwin_or_msys2,
@@ -626,3 +627,21 @@ def test_noarch_python():
     noarch = json.loads(package_has_file(fn, 'info/noarch.json').decode())
     assert 'entry_points' in noarch
     assert 'type' in noarch
+
+
+def test_skip_compile_pyc():
+    recipe = os.path.join(metadata_dir, "skip_compile_pyc")
+    fn = api.get_output_file_path(recipe)
+    api.build(recipe)
+    tf = tarfile.open(fn)
+    pyc_count = 0
+    for f in tf.getmembers():
+        filename = os.path.basename(f.name)
+        _, ext = os.path.splitext(filename)
+        basename = filename.split('.',1)[0]
+        if basename == 'skip_compile_pyc':
+            assert not ext == '.pyc', "a skip_compile_pyc .pyc was compiled: {}".format(filename)
+        if ext == '.pyc':
+            assert basename == 'compile_pyc', "an unexpected .pyc was compiled: {}".format(filename)
+            pyc_count = pyc_count+1
+    assert pyc_count == 2, "there should be 2 .pyc files, instead there were {}".format(pyc_count)
\ No newline at end of file