Repository: conda/conda-build

Issue:
Encoding issues on Windows build in the fix_staged_scripts step
Hello.

I'm trying to build the [waf package on windows](https://github.com/conda-forge/waf-feedstock/pull/4), but I'm having some trouble with ```conda_build``` in the ```fix_staged_scripts``` step. I couldn't find any similar issue. I'll follow with some details:

While building, there is a file named ```waf``` that is meant to be moved to the ```%SCRIPTS%``` folder. The problem seems to be that ```conda-build``` tries to read the file to do some checking, but fails to do so because of the file encoding (https://github.com/conda/conda-build/blob/master/conda_build/windows.py#L45)

I believe the waf file should be opened with encoding='iso-8859-1'. Making a local change replacing with the code bellow builds the package successfully, but is not a solution:

```
with open(join(scripts_dir, fn), encoding='iso-8859-1') as f:
```

Any suggestions here? I'm unsure how this case should be handled.
I'll leave some suggestions that occurred me:

1- Fix this at package level, by avoiding running into the ```fix_staged_scripts``` function, somehow. (I don't like this solution because it'll not really fix anything besides this specific package)

2- Do some try...catch with several encodings near the presented code above

3- Have a metadata somewhere (maybe in the recipe?) to handle the file encoding. Fallback to the current case if none.


Discussion:
Thanks for isolating this.  I have had a very hard time reproducing it, but your case here should work.
Using the latest `cond-build` I still get an error while trying to build `waf` using @tarcisiofischer's `meta.yaml`:

```
...
   adding waflib\extras\boost.py as waflib/extras/boost.py
   adding waflib\extras\cython.py as waflib/extras/cython.py
   adding waflib\extras\swig.py as waflib/extras/swig.py
   adding waflib\extras\use_config.py as waflib/extras/use_config.py
   adding waflib\extras\__init__.py as waflib/extras/__init__.py
Waf: Leaving directory `D:\Programming\Miniconda3\conda-bld\waf-conda_1484775768299\work\waf-waf-1.8.21\build'
'build' finished successfully (1.023s)

(D:\Programming\Miniconda3\conda-bld\waf-conda_1484775768299\_b_env) %@EXECSTR[python X:\dotfiles\\get_prompt.py]copy waf "D:\Programming\Miniconda3\conda-bld\waf-conda_1484775768299\_b_env\Scripts"
        1 arquivo(s) copiado(s).
Traceback (most recent call last):
  File "D:\Programming\Miniconda3\Scripts\conda-build-script.py", line 5, in <module>
    sys.exit(conda_build.cli.main_build.main())
  File "D:\Programming\Miniconda3\lib\site-packages\conda_build\cli\main_build.py", line 321, in main
    execute(sys.argv[1:])
  File "D:\Programming\Miniconda3\lib\site-packages\conda_build\cli\main_build.py", line 312, in execute
    noverify=args.no_verify)
  File "D:\Programming\Miniconda3\lib\site-packages\conda_build\api.py", line 89, in build
    need_source_download=need_source_download, config=config)
  File "D:\Programming\Miniconda3\lib\site-packages\conda_build\build.py", line 1460, in build_tree
    config=config)
  File "D:\Programming\Miniconda3\lib\site-packages\conda_build\build.py", line 1010, in build
    windows.build(m, build_file, config=config)
  File "D:\Programming\Miniconda3\lib\site-packages\conda_build\windows.py", line 233, in build
    fix_staged_scripts(join(config.build_prefix, 'Scripts'))
  File "D:\Programming\Miniconda3\lib\site-packages\conda_build\windows.py", line 47, in fix_staged_scripts
    line = bs4.UnicodeDammit(f.readline()).unicode_markup.lower()
  File "D:\Programming\Miniconda3\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 4240: character maps to <undefined>
```

I'm preparing a PR which definitely (I hope) fixes the issue. 😁 
 
Dang.  I really hope you have a better handle on it than I do.  I'm pretty lost on these unicode issues.  Thanks for working on it.

Resolving patch:
diff --git a/conda_build/windows.py b/conda_build/windows.py
--- a/conda_build/windows.py
+++ b/conda_build/windows.py
@@ -43,15 +43,16 @@ def fix_staged_scripts(scripts_dir):
         if not isfile(join(scripts_dir, fn)) or '.' in fn:
             continue
 
-        with open(join(scripts_dir, fn)) as f:
-            line = bs4.UnicodeDammit(f.readline()).unicode_markup.lower()
+        # read as binary file to ensure we don't run into encoding errors, see #1632
+        with open(join(scripts_dir, fn), 'rb') as f:
+            line = f.readline()
             # If it's a #!python script
-            if not (line.startswith('#!') and 'python' in line.lower()):
+            if not (line.startswith(b'#!') and b'python' in line.lower()):
                 continue
             print('Adjusting unix-style #! script %s, '
                   'and adding a .bat file for it' % fn)
             # copy it with a .py extension (skipping that first #! line)
-            with open(join(scripts_dir, fn + '-script.py'), 'w') as fo:
+            with open(join(scripts_dir, fn + '-script.py'), 'wb') as fo:
                 fo.write(f.read())
             # now create the .exe file
             copy_into(join(dirname(__file__), 'cli-%d.exe' % bits),


Test patch:
diff --git a/tests/test-recipes/metadata/_script_win_creates_exe_garbled/meta.yaml b/tests/test-recipes/metadata/_script_win_creates_exe_garbled/meta.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_script_win_creates_exe_garbled/meta.yaml
@@ -0,0 +1,13 @@
+package:
+  name: script_win_creates_exe
+  version: 1.0
+
+source:
+  path: .
+
+build:
+  script: python setup.py install
+
+requirements:
+  build:
+    - python
diff --git a/tests/test-recipes/metadata/_script_win_creates_exe_garbled/setup.py b/tests/test-recipes/metadata/_script_win_creates_exe_garbled/setup.py
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_script_win_creates_exe_garbled/setup.py
@@ -0,0 +1,5 @@
+from distutils.core import setup
+setup(name='foobar',
+      version='1.0',
+      scripts=['test-script']
+      )
diff --git a/tests/test-recipes/metadata/_script_win_creates_exe_garbled/test-script b/tests/test-recipes/metadata/_script_win_creates_exe_garbled/test-script
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_script_win_creates_exe_garbled/test-script
@@ -0,0 +1,12 @@
+#!/usr/bin/env python
+import sys
+print(sys.version)
+
+
+# binary data which might be present in a test script, to ensure we don't get encoding errors
+# while generating an executable on Windows.
+# This particular chunk was extracted from bottom of the "waf-script" file generated during build, see #1632
+#==>
+#BZh91AY&SYî?llvþÿ°#.Óÿÿÿÿÿÿÿÿÿÿÿ??Y@Â?%H4à?b<g^è?#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.ù??d\?mÏ?
+
+
diff --git a/tests/test_api_build.py b/tests/test_api_build.py
--- a/tests/test_api_build.py
+++ b/tests/test_api_build.py
@@ -743,8 +743,9 @@ def test_fix_permissions(test_config):
 
 
 @pytest.mark.skipif(not on_win, reason="windows-only functionality")
-def test_script_win_creates_exe(test_config):
-    recipe = os.path.join(metadata_dir, "_script_win_creates_exe")
+@pytest.mark.parametrize('recipe_name', ["_script_win_creates_exe", "_script_win_creates_exe_garbled"])
+def test_script_win_creates_exe(test_config, recipe_name):
+    recipe = os.path.join(metadata_dir, recipe_name)
     outputs = api.build(recipe, config=test_config)
     assert package_has_file(outputs[0], 'Scripts/test-script.exe')
     assert package_has_file(outputs[0], 'Scripts/test-script-script.py')