Repository: conda/conda-build

Issue:
OSError invalid argument mmap
Tried building a package with `conda-build` version `2.0.0` as well, but am unable to due to this error.

```
Traceback (most recent call last):
  File "/opt/conda/bin/conda-build", line 6, in <module>
    sys.exit(conda_build.cli.main_build.main())
  File "/opt/conda/lib/python3.5/site-packages/conda_build/cli/main_build.py", line 234, in main
    execute(sys.argv[1:])
  File "/opt/conda/lib/python3.5/site-packages/conda_build/cli/main_build.py", line 226, in execute
    already_built=None, config=config)
  File "/opt/conda/lib/python3.5/site-packages/conda_build/api.py", line 81, in build
    need_source_download=need_source_download, config=config)
  File "/opt/conda/lib/python3.5/site-packages/conda_build/build.py", line 942, in build_tree
    config=config)
  File "/opt/conda/lib/python3.5/site-packages/conda_build/build.py", line 700, in build
    croot=config.croot)
  File "/opt/conda/lib/python3.5/site-packages/conda_build/post.py", line 370, in post_build
    fix_shebang(f, prefix=prefix, build_python=build_python, osx_is_app=osx_is_app)
  File "/opt/conda/lib/python3.5/site-packages/conda_build/post.py", line 62, in fix_shebang
    mm = mmap.mmap(fi.fileno(), 0)
OSError: [Errno 22] Invalid argument
```

xref: https://github.com/conda-forge/conda-smithy/issues/289



Discussion:
What package?  I have not seen this error before.

Sorry should have added that. Was building `setuptools`.

And is this on Linux or Mac?  I'm guessing Linux, just want to make sure.  I have built setuptools OK on mac, but haven't tried Linux yet.

Similar issue, conda-build 2.0.1 on linux:

number of files: 2
Fixing permissions
Traceback (most recent call last):
  File "/opt/miniconda/bin/conda-build", line 6, in <module>
    sys.exit(conda_build.cli.main_build.main())
  File "/opt/miniconda/lib/python3.4/site-packages/conda_build/cli/main_build.py", line 239, in main
    execute(sys.argv[1:])
  File "/opt/miniconda/lib/python3.4/site-packages/conda_build/cli/main_build.py", line 231, in execute
    already_built=None, config=config)
  File "/opt/miniconda/lib/python3.4/site-packages/conda_build/api.py", line 83, in build
    need_source_download=need_source_download, config=config)
  File "/opt/miniconda/lib/python3.4/site-packages/conda_build/build.py", line 998, in build_tree
    config=recipe_config)
  File "/opt/miniconda/lib/python3.4/site-packages/conda_build/build.py", line 714, in build
    prefix=config.build_prefix)
  File "/opt/miniconda/lib/python3.4/site-packages/conda_build/build.py", line 352, in create_info_files
    detect_and_record_prefix_files(m, files, prefix, config)
  File "/opt/miniconda/lib/python3.4/site-packages/conda_build/build.py", line 227, in detect_and_record_prefix_files
    files_with_prefix = sorted(have_prefix_files(files, prefix))
  File "/opt/miniconda/lib/python3.4/site-packages/conda_build/build.py", line 133, in have_prefix_files
    mm = mmap.mmap(fi.fileno(), 0)
OSError: [Errno 22] Invalid argument

It may be of interest that I've set CONDA_BUILD_PATH.

@sean-abbott sorry, I'm having a hard time reproducing this.  I think I understand why it happens - somehow an empty file or link is being passed to mmap.  We do check for empty files, but perhaps not empty links.  See https://github.com/conda/conda-build/pull/1389

If you have any knowledge there, I welcome your comments.

I'm afraid I don't have any insight, I'm brand new to conda.

I was trying to set a CONDA_BLD_PATH (or whatever the variable is) in my local directory (desirable for a jenkins build; everything needs to happen in the workspace jenkins checks out in case you're running multiple builds on the same machine).  Whenever I tried to set the CONDA_BLD_PATH to anything other than default, I'd get the mmap error.

This is in a virtualbox ubuntu 16.04.1 virtualmachine, but I have a hard time believing it's because of virtualization.  The user has full permissions to both paths.

Ah, that's helpful.  So let me get the facts straight:
- Main host machine is jenkins.  What OS?  Is this the only machine, but it runs Virtualbox workers?
- You're sharing a single CONDA_BLD_PATH that is used by multiple build workers.  Are they using this path simultaneously?

No, the problem came when I was building on my local development environment. I would like to do it this way so that multiple jenkins builders do NOT share a BLD_PATH.

The idea is that devs will be able to produce builds the same way that jenkins does to reduce troubleshooting between the two.  However, when I do this, I get the mmap error.  Again, I haven't gotten it into jenkins yet because I'm still running the build locally.

Now on the virtual machine, the build path (`/code/pycanopy`) is actually mounted read/write from the host . But there are dozens of other files that conda successfully creates...

Thanks for the extra detail.  I'll fire up virtualbox and see what I can find.

Hmmm...there needs to be on where CONDA_BLD_PATH goes...it tries to copy itself to itself recursively if it's in the current build directory or anywhere below that directory.  So that's dysfunctional....

Ah, I have fought that with conda-build itself during testing, but didn't think that anyone else would encounter it.  I'll see what I can do, but this can make copying files and folders much more complicated.  This is way more complicated than it should be.

In general, I would strongly advise you to keep your notions of your source directory and you build directory well-separated.  Having them nested is just confusing, and I don't think the copying logic would ever be satisfactory.  If you want to have some single folder, I recommend something like:

```
<PROJECT_ROOT>
<PROJECT_ROOT>/src
<PROJECT_ROOT>/<CONDA_BLD_PATH>
```

I actually have that.  conda appears to be copying from the project_root rather than the src directory, though.

Is your recipe something public that you can share, or can you copy the
relevant parts into a simple test recipe?

Sorry, I'm just not sure what exactly is going on here.

On Sep 26, 2016 12:41, "Sean Abbott" notifications@github.com wrote:

> I actually have that. conda appears to be copying from the project_root
> rather than the src directory, though.
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> https://github.com/conda/conda-build/issues/1302#issuecomment-249641618,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AACV-b-q_-YdpJVM3N2oFS9hlskmiiOrks5quAO1gaJpZM4Jy43O
> .

I have a very similar issue and I may have a clue about what's going on.

Here's some background: I run a linux VM on a Mac, that share a directory from the Mac host, in which anaconda2 is installed.

`File "/home/vagrant/shared/anaconda2/lib/python2.7/site-packages/conda_build/build.py", line 133, in have_prefix_files
mm = mmap.mmap(fi.fileno(), 0)
mmap.error: [Errno 22] Invalid argument`

Problem seems to be here: https://github.com/conda/conda-build/blob/master/conda_build/build.py#L123

The filesystem is still managed by OSX, but sys.platform isn't darwin

I tried rebuilding from an anaconda install inside the VM, not on the shared folder and it worked perfectly.

yikes, that is very tricky.  I'm not familiar with any tools that Python has for determining the underlying filesystem type.  I guess the best we might be able to do is wrap the mmap in a try/catch block and show a warning, rather than crashing outright.  Thanks for helping to isolate this, @gabrielboucher 

I suspect the issue I was hitting was the same as @gabrielboucher's. Namely using a Linux VM on OS X with shared directories. In my case, this was [boot2docker](https://github.com/boot2docker/boot2docker) used from [`docker-machine`](https://github.com/docker/machine). Maybe this will help make it easier to reproduce?

I'm seen this issue also when trying to build `conda-forge` feedstocks using the `ci_suppoprt/run_docker_build.sh` script.  My set up is similar to @jakirkham, docker running on a OS X host using `docker-machine`.  If I run this same script on a Linux host the build runs fine.  

Let me know if you need more details to replicate.
Also this [SO]( http://stackoverflow.com/a/18421071 ) answer seems related. Basically looks like VirtualBox doesn't support `mmap`. See this [bug report]( https://www.virtualbox.org/ticket/819 ) for details. Is there some other way to get the intended behavior if/when `mmap` is not available?
Have recently switched to [Docker for Mac]( https://docs.docker.com/docker-for-mac/ ), which AFAIK does not use VirtualBox. However, am still having the same issues.
Could you please help us understand why `mmap` is required, @msarahan? If we can get away with using something else in its place, I suspect this problem goes away.
Looks like it was added in PR ( https://github.com/conda/conda-build/pull/866 ), correct? Maybe @brentp could weigh in on how we might fix this problem.
Had another realization on this today. It turns out this problem only occurs when the packages live in a directory that is mounted from the Host OS into the Guest OS (VM). So it has something to do with `mmap` + shared directories.

Resolving patch:
diff --git a/conda_build/build.py b/conda_build/build.py
--- a/conda_build/build.py
+++ b/conda_build/build.py
@@ -137,7 +137,10 @@ def have_prefix_files(files, prefix):
             continue
 
         fi = open(path, 'rb+')
-        mm = mmap.mmap(fi.fileno(), 0)
+        try:
+            mm = mmap.mmap(fi.fileno(), 0)
+        except OSError:
+            mm = fi
 
         mode = 'binary' if mm.find(b'\x00') != -1 else 'text'
         if mode == 'text':
diff --git a/conda_build/post.py b/conda_build/post.py
--- a/conda_build/post.py
+++ b/conda_build/post.py
@@ -56,12 +56,16 @@ def fix_shebang(f, prefix, build_python, osx_is_app=False):
     with io.open(path, encoding=locale.getpreferredencoding(), mode='r+') as fi:
         try:
             data = fi.read(100)
+            fi.seek(0)
         except UnicodeDecodeError:  # file is binary
             return
 
         # regexp on the memory mapped file so we only read it into
         # memory if the regexp matches.
-        mm = mmap.mmap(fi.fileno(), 0)
+        try:
+            mm = mmap.mmap(fi.fileno(), 0)
+        except OSError:
+            mm = fi
         m = SHEBANG_PAT.match(mm)
 
         if not (m and b'python' in m.group()):


Test patch:
diff --git a/tests/test_api_build.py b/tests/test_api_build.py
--- a/tests/test_api_build.py
+++ b/tests/test_api_build.py
@@ -18,6 +18,7 @@
 
 from binstar_client.commands import remove, show
 from binstar_client.errors import NotFound
+from pkg_resources import parse_version
 import pytest
 import yaml
 import tarfile
@@ -26,7 +27,7 @@
 from conda_build.build import VersionOrder
 from conda_build.render import finalize_metadata
 from conda_build.utils import (copy_into, on_win, check_call_env, convert_path_for_cygwin_or_msys2,
-                               package_has_file, check_output_env, conda_43)
+                               package_has_file, check_output_env)
 from conda_build.os_utils.external import find_executable
 
 from .utils import is_valid_dir, metadata_dir, fail_dir, add_mangling
@@ -646,7 +647,8 @@ def test_about_json_content(testing_metadata):
     assert 'root_pkgs' in about and about['root_pkgs']
 
 
-@pytest.mark.xfail(not conda_43(), reason="new noarch supported starting with conda 4.3")
+@pytest.mark.xfail(parse_version(conda.__version__) < parse_version("4.3.14"),
+                   reason="new noarch supported starting with conda 4.3.14")
 def test_noarch_python_with_tests(testing_config):
     recipe = os.path.join(metadata_dir, "_noarch_python_with_tests")
     api.build(recipe, config=testing_config)
@@ -843,7 +845,7 @@ def test_workdir_removal_warning_no_remove(testing_config, caplog):
 @pytest.mark.skipif(not sys.platform.startswith('linux'),
                     reason="cross compiler packages created only on Linux right now")
 @pytest.mark.xfail(True,
-                   #VersionOrder(conda.__version__) < VersionOrder('4.3.2'),
+                   # VersionOrder(conda.__version__) < VersionOrder('4.3.2'),
                    reason="not completely implemented yet")
 def test_cross_compiler(testing_workdir, testing_config, caplog):
     # TODO: testing purposes.  Package on @mingwandroid's channel.