Repository: pypa/pip

Issue:
pip10 segfaults when installing cryptography and pyopenssl
* Pip version: 10
* Python version: 2.x / 3.x
* Operating system: ubuntu at least 16.04 & 18.04 / bionic + venv

### Description:

pip10 segfaults when installing packages

### What I've run:

```
docker run -ti --rm  ubuntu:18.04 bash
#
apt update -qq
apt install -y python python-dev build-essential m4 python-pip \
bash-completion python-virtualenv libexpat1-dev:amd64 \
libffi-dev:amd64 libgmp-dev:amd64 zlib1g-dev
#
virtualenv venv && . venv/bin/activate && pip install -U --ignore-installed pip \
&& pip install -U --ignore-installed cryptography pyOpenSSL\
&& pip install -U --ignore-installed cryptography pyOpenSSL
```
Related to: https://github.com/pyca/cryptography/issues/4206


Discussion:
cc @gagaro @regilero @reaperhulk
Pinging @dstufft since he's likely the best guy for looking into this.

One possible avenue of exploration for why this is occurring is that pip vendors urllib3, which looks at the environment to automatically activate alternate SSLContext support via pyopenssl (which uses cryptography's `_openssl.so`). When `--ignore-installed` is passed pip will overwrite these packages while simultaneously using them. Of course, if that's the issue then it's mysterious why pip 9 does not have the same problem.

Edit: Quick note, this isn't a Python 2.7 specific problem or an 18.04 only issue. The segfault occurs on 2.7/3.5 in Ubuntu 16.04 as well.
i updated desc accordingly
I poked at it for a few minutes. After `apt install python-dbg`, the traceback I get is:

```
(gdb) bt
#0  PyType_Ready () at ../Objects/typeobject.c:4155
#1  0x00005555555a2eff in PyObject_Hash (v=0x7ffff2fce2f0)
    at ../Objects/object.c:1121
#2  0x0000555555727abd in _PyDict_DelItemIf () at ../Objects/dictobject.c:913
#3  0x0000555555727a61 in remove_dead_weakref.lto_priv ()
    at ../Modules/_weakref.c:31
#4  0x000055555564901a in call_function (oparg=<optimized out>, 
    pp_stack=0x7fffffffc460) at ../Python/ceval.c:4372
#5  PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#6  0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#7  0x00005555556621a9 in function_call.lto_priv ()
    at ../Objects/funcobject.c:523
#8  0x000055555563aca7 in PyObject_Call (kw=0x0, arg=0x7ffff2606cd0, 
    func=0x7ffff3e5b578) at ../Objects/abstract.c:2547
#9  PyObject_CallFunctionObjArgs () at ../Objects/abstract.c:2774
#10 0x00005555556aa453 in handle_callback.lto_priv ()
    at ../Objects/weakrefobject.c:896
#11 0x0000555555682b07 in PyObject_ClearWeakRefs ()
    at ../Objects/weakrefobject.c:941
#12 0x00005555556821ec in subtype_dealloc.lto_priv ()
    at ../Objects/typeobject.c:997
#13 0x00005555556825b4 in dict_dealloc.lto_priv.332 (mp=0x7ffff2fb04b0)
    at ../Objects/dictobject.c:1086
#14 subtype_dealloc.lto_priv () at ../Objects/typeobject.c:1035
#15 0x000055555565c558 in frame_dealloc.lto_priv ()
    at ../Objects/frameobject.c:494
#16 0x000055555564dd93 in fast_function (nk=<optimized out>, 
    na=<optimized out>, n=1, pp_stack=0x7fffffffc900, func=<optimized out>)
    at ../Python/ceval.c:4459
#17 call_function (oparg=<optimized out>, pp_stack=0x7fffffffc900)
    at ../Python/ceval.c:4392
#18 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#19 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#20 0x000055555564e7d3 in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffcae0, func=<optimized out>)
    at ../Python/ceval.c:4467
#21 call_function (oparg=<optimized out>, pp_stack=0x7fffffffcae0)
    at ../Python/ceval.c:4392
#22 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#23 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#24 0x000055555564e7d3 in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffccc0, func=<optimized out>)
    at ../Python/ceval.c:4467
#25 call_function (oparg=<optimized out>, pp_stack=0x7fffffffccc0)
---Type <return> to continue, or q <return> to quit---
    at ../Python/ceval.c:4392
#26 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#27 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#28 0x000055555564e7d3 in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffcea0, func=<optimized out>)
    at ../Python/ceval.c:4467
#29 call_function (oparg=<optimized out>, pp_stack=0x7fffffffcea0)
    at ../Python/ceval.c:4392
#30 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#31 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#32 0x000055555564e7d3 in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffd080, func=<optimized out>)
    at ../Python/ceval.c:4467
#33 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd080)
    at ../Python/ceval.c:4392
#34 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#35 0x000055555564dd72 in fast_function (nk=<optimized out>, 
    na=<optimized out>, n=1, pp_stack=0x7fffffffd190, func=<optimized out>)
    at ../Python/ceval.c:4457
#36 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd190)
    at ../Python/ceval.c:4392
#37 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#38 0x000055555564dd72 in fast_function (nk=<optimized out>, 
    na=<optimized out>, n=1, pp_stack=0x7fffffffd2a0, func=<optimized out>)
    at ../Python/ceval.c:4457
#39 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd2a0)
    at ../Python/ceval.c:4392
#40 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#41 0x000055555564dd72 in fast_function (nk=<optimized out>, 
    na=<optimized out>, n=1, pp_stack=0x7fffffffd3b0, func=<optimized out>)
    at ../Python/ceval.c:4457
#42 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd3b0)
    at ../Python/ceval.c:4392
#43 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#44 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#45 0x00005555556621a9 in function_call.lto_priv ()
    at ../Objects/funcobject.c:523
#46 0x000055555567a78e in PyObject_Call (kw=0x0, arg=0x7ffff2452e68, 
    func=0x7ffff3b14578) at ../Objects/abstract.c:2547
#47 instancemethod_call.lto_priv () at ../Objects/classobject.c:2600
#48 0x000055555563a982 in PyObject_Call (kw=0x0, arg=0x7ffff1154c30, 
    func=0x7ffff3a12870) at ../Objects/abstract.c:2547
#49 PyObject_CallFunctionObjArgs () at ../Objects/abstract.c:2774
#50 0x000055555564b412 in PyEval_EvalFrameEx () at ../Python/ceval.c:2970
---Type <return> to continue, or q <return> to quit---
#51 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#52 0x000055555564e7d3 in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffdb20, func=<optimized out>)
    at ../Python/ceval.c:4467
#53 call_function (oparg=<optimized out>, pp_stack=0x7fffffffdb20)
    at ../Python/ceval.c:4392
#54 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#55 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#56 0x000055555564e7d3 in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffdd00, func=<optimized out>)
    at ../Python/ceval.c:4467
#57 call_function (oparg=<optimized out>, pp_stack=0x7fffffffdd00)
    at ../Python/ceval.c:4392
#58 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#59 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#60 0x000055555564e24e in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffdee0, func=<optimized out>)
    at ../Python/ceval.c:4467
#61 call_function (oparg=<optimized out>, pp_stack=0x7fffffffdee0)
    at ../Python/ceval.c:4392
#62 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#63 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#64 0x000055555564c2fb in PyEval_EvalCode () at ../Python/ceval.c:669
#65 exec_statement (locals=0x7ffff7f81168, globals=0x7ffff7f81168, 
    prog=<optimized out>, f=<optimized out>) at ../Python/ceval.c:5090
#66 PyEval_EvalFrameEx () at ../Python/ceval.c:2118
#67 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#68 0x000055555564e24e in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffe2a0, func=<optimized out>)
    at ../Python/ceval.c:4467
#69 call_function (oparg=<optimized out>, pp_stack=0x7fffffffe2a0)
    at ../Python/ceval.c:4392
#70 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#71 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#72 0x00005555556621a9 in function_call.lto_priv ()
    at ../Objects/funcobject.c:523
#73 0x000055555563233e in PyObject_Call () at ../Objects/abstract.c:2547
#74 0x00005555556a3bf0 in RunModule.lto_priv.1251 () at ../Modules/main.c:197
#75 0x0000555555620406 in Py_Main () at ../Modules/main.c:592
#76 0x00007ffff7a05b97 in __libc_start_main (main=0x55555561fef0 <main>, 
    argc=8, argv=0x7fffffffe698, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffe688)
    at ../csu/libc-start.c:310
#77 0x000055555561fe0a in _start ()
```

or 

```
(gdb) bt
#0  0x000055555562a174 in visit_decref () at ../Modules/gcmodule.c:360
#1  0x000055555562a334 in dict_traverse.lto_priv ()
    at ../Objects/dictobject.c:2198
#2  0x00005555556290e5 in subtract_refs () at ../Modules/gcmodule.c:385
#3  collect.lto_priv () at ../Modules/gcmodule.c:925
#4  0x0000555555684b61 in PyGC_Collect () at ../Modules/gcmodule.c:1440
#5  0x0000555555683e23 in Py_Finalize () at ../Python/pythonrun.c:455
#6  0x00005555556a7bf8 in Py_Exit (sts=sts@entry=0)
    at ../Python/pythonrun.c:1792
#7  0x00005555556a5341 in handle_system_exit () at ../Python/pythonrun.c:1160
#8  0x00005555556a4c83 in PyErr_PrintEx () at ../Python/pythonrun.c:1170
#9  0x00005555556a3bfd in RunModule.lto_priv.1251 () at ../Modules/main.c:199
#10 0x0000555555620406 in Py_Main () at ../Modules/main.c:592
#11 0x00007ffff7a05b97 in __libc_start_main (main=0x55555561fef0 <main>, 
    argc=8, argv=0x7fffffffe698, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffe688)
    at ../csu/libc-start.c:310
#12 0x000055555561fe0a in _start ()
```

So whatever's going wrong seems to be going wrong during garbage collection. Possibly there is some `__del__` method doing something nasty?

I also tried running it under `rr`, but `rr replay` itself segfaulted! It [looks like](https://github.com/mozilla/rr/wiki/Docker) I need to restart the container with `--cap-add=SYS_PTRACE --security-opt seccomp=unconfined`, but that would require rebuilding the container and I've run out of time for this right now.
> Of course, if that's the issue then it's mysterious why pip 9 does not have the same problem.

That's not that mysterious, pip 9 explicitly patched requests/urllib3 to ignore pyopenssl etc and only use the standard library, and pip 10 enabled using pyopenssl etc on non-Windows platforms to enable people to get better TLS if possible. If that's causing issues we can probably just go back to disabling anything but stdlib TLS.
Hello,
I have the same issue with Debian 8 on ARM.
BTW, it's not in general true that replacing files that are in use causes problems on Linux. So long as pip replaces files by creating new files and `rename`ing them over the original, and pip isn't importing new extension modules afterwards, then all should be fine. (This is how `apt` can upgrade your system without requiring a reboot every time.) So there's something more subtle going on than just "oh well this is impossible".
@njsmith yeah I figured it shouldn't matter (unlinking a file doesn't unmap it from memory and all), so I'm definitely only stating that this occurs when it replaces existing files. I'm not quite sure where to go in debugging this though.
> If that's causing issues we can probably just go back to disabling anything but stdlib TLS.

+1

Okay here's the issue (huge thanks to @geofft for essentially solving this problem solely from being given an incomplete summary):

When pyOpenSSL is present in an environment pip 10's vendored urllib3 will automatically use it for generating SSLContexts. pyOpenSSL uses cryptography, which has a native shared object called _openssl.so that Python dlopens. dlopen mmaps the file and then methods are called as needed.

When using `pip install --ignore-installed` to install a wheel pip uses the `clobber` method, which ultimately calls `shutil.copyfile`. This function truncates the file and then writes the new data ([opening a file to write truncates it](https://github.com/python/cpython/blob/7ec8f28656ea9d84048e9b5655375c6a74a59f53/Lib/shutil.py#L121)). This is problematic because we've mmap'd the file and the process expects to be able to call symbols that no longer exist. When it attempts to do so (and said symbol isn't already resident in "real" memory) it segfaults.

The easy fix here is to add this to the clobber method:

```diff
+               if os.path.exists(destfile):
+                  os.unlink(destfile)
                shutil.copyfile(srcfile, destfile)
```

This will unlink the file (if it exists) and write a new one. Since the file was mmap'd Python will continue to be able to access the data it needs and exit normally.

Resolving patch:
diff --git a/src/pip/_internal/wheel.py b/src/pip/_internal/wheel.py
--- a/src/pip/_internal/wheel.py
+++ b/src/pip/_internal/wheel.py
@@ -285,6 +285,17 @@ def clobber(source, dest, is_base, fixer=None, filter=None):
                 # uninstalled.
                 ensure_dir(destdir)
 
+                # copyfile (called below) truncates the destination if it
+                # exists and then writes the new contents. This is fine in most
+                # cases, but can cause a segfault if pip has loaded a shared
+                # object (e.g. from pyopenssl through its vendored urllib3)
+                # Since the shared object is mmap'd an attempt to call a
+                # symbol in it will then cause a segfault. Unlinking the file
+                # allows writing of new contents while allowing the process to
+                # continue to use the old copy.
+                if os.path.exists(destfile):
+                    os.unlink(destfile)
+
                 # We use copyfile (not move, copy, or copy2) to be extra sure
                 # that we are not moving directories over (copyfile fails for
                 # directories) as well as to ensure that we are not copying


Test patch: