Repository: pex-tool/pex

Issue:
Pex resources leak while bootstrapping pants
```
Successfully installed PyYAML-5.3.1 ansicolors-1.1.8 certifi-2020.6.20 cffi-1.14.3 chardet-3.0.4 cryptography-3.1.1 dataclasses-0.6 fasteners-0.15 idna-2.10 monotonic-1.5 packaging-20.4 pantsbuild.pants-2.0.0b1 pex-2.1.16 psutil-5.7.0 pyOpenSSL-19.1.0 pycparser-2.20 pyparsing-2.4.7 python-Levenshtein-0.12.0 requests-2.24.0 setproctitle-1.1.10 setuptools-50.3.0 six-1.15.0 toml-0.10.1 typed-ast-1.4.1 typing-extensions-3.7.4.2 urllib3-1.25.10 www-authenticate-0.9.2
New virtual environment successfully created at /home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38.
15:39:17.38 [INFO] Resolving new plugins...:
  networkx==2.4
  python-dateutil
15:39:17.48 [WARN] /home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/third_party/__init__.py:382: ResourceWarning: unclosed file <_io.BufferedReader name='/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/pex_builder.py'>
  shutil.copyfileobj(resource_stream(module, src_entry), fp)
15:39:17.48 [WARN] /home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/third_party/__init__.py:382: ResourceWarning: unclosed file <_io.BufferedReader name='/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/bootstrap.py'>
  shutil.copyfileobj(resource_stream(module, src_entry), fp)
15:39:17.48 [WARN] /home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/third_party/__init__.py:382: ResourceWarning: unclosed file <_io.BufferedReader name='/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/network_configuration.py'>
  shutil.copyfileobj(resource_stream(module, src_entry), fp)
15:39:17.49 [WARN] /home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/third_party/__init__.py:382: ResourceWarning: unclosed file <_io.BufferedReader name='/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/pex_info.py'>
  shutil.copyfileobj(resource_stream(module, src_entry), fp)
15:39:17.49 [WARN] /home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/third_party/__init__.py:382: ResourceWarning: unclosed file <_io.BufferedReader name='/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/dist_metadata.py'>
  shutil.copyfileobj(resource_stream(module, src_entry), fp)
15:39:17.49 [WARN] /home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/third_party/__init__.py:382: ResourceWarning: unclosed file <_io.BufferedReader name='/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/interpreter_constraints.py'>
  shutil.copyfileobj(resource_stream(module, src_entry), fp)
15:39:17.49 [WARN] /home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/third_party/__init__.py:382: ResourceWarning: unclosed file <_io.BufferedReader name='/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/testing.py'>
  shutil.copyfileobj(resource_stream(module, src_entry), fp)
15:39:17.49 [WARN] /home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/third_party/__init__.py:382: ResourceWarning: unclosed file <_io.BufferedReader name='/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b1_py38/lib/python3.8/site-packages/pex/pip.py'>
  shutil.copyfileobj(resource_stream(module, src_entry), fp)
```


Discussion:
https://pantsbuild.slack.com/archives/C0D7TNJHL/p1601484678103800
@jsirois @benjyw @stuhood 
I don't think this is new,  it is likely that this change  https://github.com/pantsbuild/pants/pull/10845  now causes pants/pex to log this.

Agreed - this has been broken from day 1. resource_stream returns a file-like object that simply needs to be closed.

Resolving patch:
diff --git a/pex/third_party/__init__.py b/pex/third_party/__init__.py
--- a/pex/third_party/__init__.py
+++ b/pex/third_party/__init__.py
@@ -11,6 +11,7 @@
 import sys
 import zipfile
 from collections import OrderedDict, namedtuple
+from contextlib import closing
 
 
 # NB: All pex imports are performed lazily to play well with the un-imports performed by both the
@@ -379,7 +380,8 @@ def recursive_copy(srcdir, dstdir):
                     recursive_copy(src_entry, dst_entry)
                 elif not entry_name.endswith(".pyc"):
                     with open(dst_entry, "wb") as fp:
-                        shutil.copyfileobj(resource_stream(module, src_entry), fp)
+                        with closing(resource_stream(module, src_entry)) as resource:
+                            shutil.copyfileobj(resource, fp)
 
         pex_path = os.path.join(vendor.VendorSpec.ROOT, "pex")
         with _tracer().timed("Hashing pex"):


Test patch: