Repository: pex-tool/pex

Issue:
Pex fails to consistently use https_proxy + --cert
No self-contained repro yet, but a user reports the following:

In a container where the ambient REQUESTS_CA_BUNDLE points to a host-system self-signed cert authority:
```
$ docker run --rm -it -e HTTP_PROXY -e HTTPS_PROXY -v $REQUESTS_CA_BUNDLE:/ca-cert.crt -v $(pwd):/app python:3.7 bash
$ pip install --cert /ca-cert.crt pex
$ pex -vvvvvvvvv --cert /ca-cert.crt black setuptools
...
pex: Building pex :: Resolving distributions (['black', 'setuptools']) :: Resolving for:
pex: Spawning a maximum of 12 parallel jobs to process:r/local/bin/python3.7', PythonIdentity('/usr/local/bin/python', 'cp37', 'cp37m', 'manylinux2014_x86_64', (3, 7, 9))))
  DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.7', PythonIdentity('/usr/local/bin/python', 'cp37', 'cp37m', 'manylinux2014_x86_64', (3, 7, 9))))
pex: Hashing pex: 39.5ms
pex: Isolating pex: 97.5ms
pex:   Extracting pex to /root/.pex/isolated/a8ebd39cfba0316d0416132e488ffc14d28893b5: 97.0ms
pex: Vendoring packaging from VendorSpec(key='packaging', requirement='packaging==19.2', rewrite=True, constrain=True) @ /usr/local/lib/python3.7/site-packages/pex/vendor/_vendored/packaging
pex: Vendoring pyparsing from VendorSpec(key='packaging', requirement='packaging==19.2', rewrite=True, constrain=True) @ /usr/local/lib/python3.7/site-packages/pex/vendor/_vendored/packaging
pex: Vendoring pkg_resources from VendorSpec(key='setuptools', requirement='setuptools==42.0.2', rewrite=True, constrain=True) @ /usr/local/lib/python3.7/site-packages/pex/vendor/_vendored/setuptools
pex: Executing: /usr/local/bin/python3.7 -s -E /tmp/tmplte70n5r
pex: PEX.run invoking /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5 --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -vvv --cache-dir /root/.pex download --dest /tmp/tmpt3ltlh3v/usr.local.bin.python3.7 --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --cert /ca-cert.crt black setuptools
pex: Installed VendorImporter(root='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', importables=(_Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: Installed VendorImporter(root='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', importables=(_Importable(module='compiler', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='distribution_target', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='resolver', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='variables', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='executor', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='version', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_info', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='tracer', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='environment', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='testing', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='compatibility', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='bootstrap', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='network_configuration', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='__main__', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='util', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_bootstrapper', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='requirements', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_builder', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='jobs', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='typing', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pip', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='interpreter_constraints', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='interpreter', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='finders', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='platforms', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='orderedset', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='common', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_warnings', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='dist_metadata', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='vendor', is_pkg=True, path='pex', prefix='_pex'), _Importable(module='third_party', is_pkg=True, path='pex', prefix='_pex')))
pex: pex.third_party.packaging imported via _Loader(module_name='pex.third_party.packaging', vendor_module_name='pex.vendor._vendored.packaging.packaging')
pex: pex.third_party.pyparsing imported via _Loader(module_name='pex.third_party.pyparsing', vendor_module_name='pex.vendor._vendored.packaging.pyparsing')
pex: pex.third_party.pkg_resources imported via _Loader(module_name='pex.third_party.pkg_resources', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources')
pex: pex.third_party.pkg_resources.extern imported via _Loader(module_name='pex.third_party.pkg_resources.extern', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern')
pex: pex.third_party.pkg_resources.extern.six imported via _Loader(module_name='pex.third_party.pkg_resources.extern.six', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.six')
pex: pex.third_party.pkg_resources.extern.six.moves imported via _Loader(module_name='pex.third_party.pkg_resources.extern.six.moves', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.six.moves')
pex: pex.third_party.pkg_resources.extern.packaging imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging')
pex: pex.third_party.pkg_resources.extern.packaging.version imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging.version', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.version')
pex: pex.third_party.pkg_resources.extern.packaging.specifiers imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging.specifiers', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.specifiers')
pex: pex.third_party.pkg_resources.extern.pyparsing imported via _Loader(module_name='pex.third_party.pkg_resources.extern.pyparsing', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.pyparsing')
pex: pex.third_party.pkg_resources.extern.six.moves.urllib imported via _Loader(module_name='pex.third_party.pkg_resources.extern.six.moves.urllib', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.six.moves.urllib')
pex: pex.third_party.pkg_resources.extern.packaging.requirements imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging.requirements', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.requirements')
pex: pex.third_party.pkg_resources.extern.packaging.markers imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging.markers', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.markers')
pex: pex.third_party.pkg_resources imported via _Loader(module_name='pex.third_party.pkg_resources', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources')
pex: pex.third_party.packaging.specifiers imported via _Loader(module_name='pex.third_party.packaging.specifiers', vendor_module_name='pex.vendor._vendored.packaging.packaging.specifiers')
pex: Using the current interpreter /usr/local/bin/python3.7 since no constraints have been specified and PYTHONPATH is not set.
pex: Selecting runtime interpreter: 0.1ms
pex: Found site-library: /usr/local/lib/python3.7/site-packages
pex: Found .pth file: /usr/local/lib/python3.7/site-packages/distutils-precedence.pth
pex: Not a tainted path element: /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap
pex: Not a tainted path element: /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5
pex: Not a tainted path element: /usr/local/lib/python37.zip
pex: Not a tainted path element: /usr/local/lib/python3.7
pex: Not a tainted path element: /usr/local/lib/python3.7/lib-dynload
pex: Tainted path element: /usr/local/lib/python3.7/site-packages
pex: Scrubbing from user site: /root/.local/lib/python3.7/site-packages
pex: Scrubbing from site-packages: /usr/local/lib/python3.7/site-packages
pex: Dropping _distutils_hack
pex: Dropping setuptools
pex: Dropping _distutils_hack.override
pex: Dropping setuptools._distutils
pex: Dropping distutils
pex: Dropping distutils.core
pex: Dropping distutils.debug
pex: Dropping distutils.errors
pex: Dropping distutils.dist
pex: Dropping distutils.fancy_getopt
pex: Dropping distutils.util
pex: Dropping distutils.dep_util
pex: Dropping distutils.spawn
pex: Dropping distutils.log
pex: Dropping distutils.py35compat
pex: Dropping distutils.cmd
pex: Dropping distutils.dir_util
pex: Dropping distutils.file_util
pex: Dropping distutils.archive_util
pex: Dropping distutils.config
pex: Dropping distutils.extension
pex: Dropping setuptools._deprecation_warning
pex: Dropping setuptools.version
pex: Dropping pkg_resources
pex: Dropping pkg_resources.extern
pex: Dropping pkg_resources._vendor
pex: Dropping pkg_resources._vendor.appdirs
pex: Dropping pkg_resources.extern.appdirs
pex: Dropping pkg_resources._vendor.packaging
pex: Dropping pkg_resources._vendor.packaging.__about__
pex: Dropping pkg_resources.extern.packaging
pex: Dropping pkg_resources.extern.packaging.version
pex: Dropping pkg_resources.extern.packaging._structures
pex: Dropping pkg_resources.extern.packaging._typing
pex: Dropping pkg_resources.extern.packaging.specifiers
pex: Dropping pkg_resources.extern.packaging._compat
pex: Dropping pkg_resources.extern.packaging.utils
pex: Dropping pkg_resources.extern.packaging.requirements
pex: Dropping pkg_resources._vendor.pyparsing
pex: Dropping pkg_resources.extern.pyparsing
pex: Dropping pkg_resources.extern.packaging.markers
pex: Dropping setuptools.extension
pex: Dropping setuptools.monkey
pex: Dropping distutils.filelist
pex: Dropping setuptools.dist
pex: Dropping distutils.version
pex: Dropping setuptools.extern
pex: Dropping setuptools._vendor
pex: Dropping setuptools._vendor.packaging
pex: Dropping setuptools._vendor.packaging.__about__
pex: Dropping setuptools.extern.packaging
pex: Dropping setuptools._vendor.ordered_set
pex: Dropping setuptools.extern.ordered_set
pex: Dropping setuptools.windows_support
pex: Dropping setuptools.config
pex: Dropping setuptools.extern.packaging.version
pex: Dropping setuptools.extern.packaging._structures
pex: Dropping setuptools.extern.packaging._typing
pex: Dropping setuptools.extern.packaging.specifiers
pex: Dropping setuptools.extern.packaging._compat
pex: Dropping setuptools.extern.packaging.utils
pex: Dropping setuptools.depends
pex: Dropping setuptools._imp
pex: Dropping setuptools.py34compat
pex: Dropping setuptools.msvc
pex: Dropping distutils.ccompiler
pex: Dropping distutils.sysconfig
pex: Dropping distutils.text_file
pex: New sys.path: ['/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload']
pex: E: tags for 'manylinux2014_x86_64' x PythonInterpreter('/usr/local/bin/python3.7', PythonIdentity('/usr/local/bin/python3.7', 'cp37', 'cp37m', 'manylinux2014_x86_64', (3, 7, 9))) -> frozenset({<cp37-cp37m-manylinux1_x86_64 @ 140329298470704>, <py30-none-manylinux2014_x86_64 @ 140329298487648>, <cp37-none-manylinux2010_x86_64 @ 140329298471584>, <py30-none-any @ 140329298488928>, <cp35-abi3-manylinux2014_x86_64 @ 140329298472544>, <py34-none-linux_x86_64 @ 140329298485888>, <cp36-abi3-linux_x86_64 @ 140329298472464>, <py35-none-any @ 140329298488528>, <py35-none-linux_x86_64 @ 140329298485328>, <py33-none-linux_x86_64 @ 140329298486448>, <py37-none-manylinux1_x86_64 @ 140329298483392>, <py35-none-manylinux2014_x86_64 @ 140329298484752>, <cp36-abi3-manylinux2014_x86_64 @ 140329298471984>, <cp37-none-any @ 140329298488208>, <py32-none-manylinux1_x86_64 @ 140329298486848>, <cp37-none-manylinux2014_x86_64 @ 140329298471424>, <py36-none-manylinux1_x86_64 @ 140329298484512>, <py30-none-linux_x86_64 @ 140329298488128>, <py34-none-manylinux1_x86_64 @ 140329298485728>, <cp37-abi3-manylinux1_x86_64 @ 140329298471184>, <cp36-abi3-manylinux1_x86_64 @ 140329298472304>, <cp35-abi3-manylinux1_x86_64 @ 140329298472864>, <cp37-none-linux_x86_64 @ 140329298471904>, <py32-none-any @ 140329298488768>, <py37-none-manylinux2014_x86_64 @ 140329298483072>, <cp37-cp37m-linux_x86_64 @ 140329298470384>, <cp34-abi3-linux_x86_64 @ 140329298481872>, <py32-none-manylinux2014_x86_64 @ 140329298486528>, <py37-none-any @ 140329298488288>, <cp37-abi3-manylinux2014_x86_64 @ 140329298470864>, <py31-none-linux_x86_64 @ 140329298487568>, <py31-none-any @ 140329298488848>, <py34-none-any @ 140329298488608>, <py31-none-manylinux2014_x86_64 @ 140329298487088>, <py34-none-manylinux2014_x86_64 @ 140329298485408>, <cp37-cp37m-manylinux2010_x86_64 @ 140329298470544>, <py36-none-linux_x86_64 @ 140329298484672>, <cp37-cp37m-manylinux2014_x86_64 @ 140329298389504>, <py30-none-manylinux1_x86_64 @ 140329298487968>, <py31-none-manylinux2010_x86_64 @ 140329298487248>, <py3-none-manylinux2010_x86_64 @ 140329298483792>, <py35-none-manylinux1_x86_64 @ 140329298485072>, <py33-none-manylinux2010_x86_64 @ 140329298486128>, <py36-none-any @ 140329298488448>, <cp33-abi3-manylinux1_x86_64 @ 140329298482272>, <py33-none-any @ 140329298488688>, <cp32-abi3-manylinux2010_x86_64 @ 140329298482672>, <cp37-none-manylinux1_x86_64 @ 140329298471744>, <py36-none-manylinux2014_x86_64 @ 140329298484192>, <py33-none-manylinux2014_x86_64 @ 140329298485968>, <cp34-abi3-manylinux1_x86_64 @ 140329298481712>, <py36-none-manylinux2010_x86_64 @ 140329298484352>, <cp37-abi3-linux_x86_64 @ 140329298471344>, <cp32-abi3-linux_x86_64 @ 140329298482992>, <cp35-abi3-linux_x86_64 @ 140329298481312>, <cp32-abi3-manylinux2014_x86_64 @ 140329298482512>, <cp36-abi3-manylinux2010_x86_64 @ 140329298472144>, <cp35-abi3-manylinux2010_x86_64 @ 140329298472704>, <py37-none-manylinux2010_x86_64 @ 140329298483232>, <py3-none-manylinux1_x86_64 @ 140329298483952>, <cp33-abi3-linux_x86_64 @ 140329298482432>, <cp34-abi3-manylinux2010_x86_64 @ 140329298481552>, <py32-none-manylinux2010_x86_64 @ 140329298486688>, <cp33-abi3-manylinux2014_x86_64 @ 140329298481952>, <py3-none-any @ 140329298488368>, <py34-none-manylinux2010_x86_64 @ 140329298485568>, <py3-none-manylinux2014_x86_64 @ 140329298483632>, <py32-none-linux_x86_64 @ 140329298487008>, <cp37-abi3-manylinux2010_x86_64 @ 140329298471024>, <cp33-abi3-manylinux2010_x86_64 @ 140329298482112>, <py37-none-linux_x86_64 @ 140329298483552>, <py33-none-manylinux1_x86_64 @ 140329298486288>, <cp34-abi3-manylinux2014_x86_64 @ 140329298481392>, <cp32-abi3-manylinux1_x86_64 @ 140329298482832>, <py30-none-manylinux2010_x86_64 @ 140329298487808>, <py3-none-linux_x86_64 @ 140329298484112>, <py35-none-manylinux2010_x86_64 @ 140329298484912>, <py31-none-manylinux1_x86_64 @ 140329298487408>})
pex: Activating PEX virtual environment from /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5: 10.8ms
pex:   Searching dependency cache: /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps: 1.4ms
pex:     Adding setuptools 42.0.2: 0.1ms
pex:     Adding wheel 0.33.6: 0.1ms
pex:     Adding pip 20.0.dev0: 0.1ms
pex:   Resolving pip from [Requirement.parse('pip==20.0.dev0')]: 2.0ms
pex:   Resolving setuptools from [Requirement.parse('setuptools==42.0.2')]: 2.7ms
pex:   Resolving wheel from [Requirement.parse('wheel==0.33.6')]: 1.9ms
pex:   Activating pip 20.0.dev0: 0.2ms
pex:     Adding sitedir: 0.1ms
pex:   Activating setuptools 42.0.2: 0.3ms
pex:     Adding sitedir: 0.2ms
pex:   Activating wheel 0.33.6: 0.3ms
pex:     Adding sitedir: 0.2ms
pex: Bootstrap complete, performing final sys.path modifications...
pex: Not uninstalling VendorImporter(root='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', importables=(_Importable(module='compiler', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='distribution_target', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='resolver', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='variables', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='executor', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='version', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_info', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='tracer', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='environment', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='testing', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='compatibility', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='bootstrap', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='network_configuration', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='__main__', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='util', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_bootstrapper', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='requirements', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_builder', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='jobs', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='typing', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pip', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='interpreter_constraints', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='interpreter', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='finders', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='platforms', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='orderedset', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='common', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_warnings', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='dist_metadata', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='vendor', is_pkg=True, path='pex', prefix='_pex'), _Importable(module='third_party', is_pkg=True, path='pex', prefix='_pex')))
pex: un-imported pex.third_party.packaging
pex: un-imported pex.third_party.packaging.specifiers
pex: un-imported pex.vendor._vendored.packaging.packaging
pex: un-imported pex.vendor._vendored.packaging.packaging.__about__
pex: un-imported pex.vendor._vendored.packaging.packaging._compat
pex: un-imported pex.vendor._vendored.packaging.packaging._structures
pex: un-imported pex.vendor._vendored.packaging.packaging.markers
pex: un-imported pex.vendor._vendored.packaging.packaging.specifiers
pex: un-imported pex.vendor._vendored.packaging.packaging.tags
pex: un-imported pex.vendor._vendored.packaging.packaging.version
pex: un-imported pex.third_party.pyparsing
pex: un-imported pex.vendor._vendored.packaging.pyparsing
pex: un-imported pex.third_party.pkg_resources
pex: un-imported pex.third_party.pkg_resources.extern
pex: un-imported pex.third_party.pkg_resources.extern.packaging
pex: un-imported pex.third_party.pkg_resources.extern.packaging.markers
pex: un-imported pex.third_party.pkg_resources.extern.packaging.requirements
pex: un-imported pex.third_party.pkg_resources.extern.packaging.specifiers
pex: un-imported pex.third_party.pkg_resources.extern.packaging.version
pex: un-imported pex.third_party.pkg_resources.extern.pyparsing
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources._vendor
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources._vendor.packaging.__about__
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.appdirs
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging._compat
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging._structures
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.markers
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.requirements
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.specifiers
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.version
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.pyparsing
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.py31compat
pex: Uninstalled VendorImporter(root='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', importables=(_Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: Demoting code from Bootstrap(sys_path_entry='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap')
pex: un-imported <module 'pex.version' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/version.py'>
pex: un-imported <module 'pex.vendor._vendored.setuptools' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/vendor/_vendored/setuptools/__init__.py'>
pex: un-imported <module 'pex.vendor._vendored.packaging' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/vendor/_vendored/packaging/__init__.py'>
pex: un-imported <module 'pex.vendor._vendored' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/vendor/_vendored/__init__.py'>
pex: un-imported <module 'pex.vendor' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/vendor/__init__.py'>
pex: un-imported <module 'pex.variables' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/variables.py'>
pex: un-imported <module 'pex.util' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/util.py'>
pex: un-imported <module 'pex.typing' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/typing.py'>
pex: un-imported <module 'pex.tracer' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/tracer.py'>
pex: un-imported <module 'pex.third_party' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/third_party/__init__.py'>
pex: un-imported <module 'pex.platforms' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/platforms.py'>
pex: un-imported <module 'pex.pip' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pip.py'>
pex: un-imported <module 'pex.pex_warnings' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex_warnings.py'>
pex: un-imported <module 'pex.pex_info' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex_info.py'>
pex: un-imported <module 'pex.pex_builder' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex_builder.py'>
pex: un-imported <module 'pex.pex_bootstrapper' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex_bootstrapper.py'>
pex: un-imported <module 'pex.pex' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex.py'>
pex: un-imported <module 'pex.orderedset' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/orderedset.py'>
pex: un-imported <module 'pex.network_configuration' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/network_configuration.py'>
pex: un-imported <module 'pex.jobs' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/jobs.py'>
pex: un-imported <module 'pex.interpreter_constraints' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/interpreter_constraints.py'>
pex: un-imported <module 'pex.interpreter' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/interpreter.py'>
pex: un-imported <module 'pex.finders' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/finders.py'>
pex: un-imported <module 'pex.executor' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/executor.py'>
pex: un-imported <module 'pex.environment' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/environment.py'>
pex: un-imported <module 'pex.distribution_target' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/distribution_target.py'>
pex: un-imported <module 'pex.dist_metadata' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/dist_metadata.py'>
pex: un-imported <module 'pex.compiler' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/compiler.py'>
pex: un-imported <module 'pex.compatibility' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/compatibility.py'>
pex: un-imported <module 'pex.common' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/common.py'>
pex: un-imported <module 'pex.bootstrap' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/bootstrap.py'>
pex: un-imported <module 'pex' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/__init__.py'>
pex: Re-imported pex from ['/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex']
pex: PYTHONPATH contains:
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5
pex:   * /usr/local/lib/python37.zip
pex:     /usr/local/lib/python3.7
pex:     /usr/local/lib/python3.7/lib-dynload
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/setuptools
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/wheel
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
Created temporary directory: /tmp/pip-req-tracker-iad7wt7d
Initialized build tracking at /tmp/pip-req-tracker-iad7wt7d
Created build tracker: /tmp/pip-req-tracker-iad7wt7d
Entered build tracker: /tmp/pip-req-tracker-iad7wt7d
Created temporary directory: /tmp/pip-download-42q1y0wc
1 location(s) to search for versions of black:
* https://pypi.org/simple/black/
Fetching project page and analyzing links: https://pypi.org/simple/black/
Getting page https://pypi.org/simple/black/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/black/" in the cache
No cache entry available
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/black/ HTTP/1.1" 200 4035
Updating cache with response from "https://pypi.org/simple/black/"
Caching due to etag
  Found link https://files.pythonhosted.org/packages/22/5a/ebe5881cbf07435ea1ad76119ee142f42fb7a333d562c202db015b959ae7/black-18.3a0-py3-none-any.whl#sha256=7183263650ba3071034e90b40a1ea74abccbd32cf525cef6d7914479dbe7f2fb (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a0
  Found link https://files.pythonhosted.org/packages/83/a2/fae3ddb516a52a0f37e6e2ef3bd789b6d093d2c7015a41a41bff30f39897/black-18.3a0.tar.gz#sha256=0461c7a52b5beb378936bf642753dec7a45305c96c6129d540b9c53227121a5a (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a0
  Found link https://files.pythonhosted.org/packages/61/b4/3a39dcf59342f53680c59a5e44c0b172ea5d6470a5aaba8ae3cdbfccc1e7/black-18.3a1-py3-none-any.whl#sha256=04edc87b51b34a7aa093f39bf207e2ebc3612b26f2c79fdf8ff13b2d60622c9a (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a1
  Found link https://files.pythonhosted.org/packages/1b/4d/ff18ef3018083196b85c843387dfa6a1faf376e8f662f74634bad84092d5/black-18.3a1.tar.gz#sha256=c9662827051587b732ecb4994a268f218eda896137b3ea6b8af8510682438b79 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a1
  Found link https://files.pythonhosted.org/packages/83/12/450fb640290d1499ae3922f95190b66b36918656c68376517a8477c6c4c9/black-18.3a2-py36-none-any.whl#sha256=e18daeede445c8d78da14734fa87015ba94ab3d2320792fdf31d95ff5a8eb8eb (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a2
  Found link https://files.pythonhosted.org/packages/f7/2b/359a321cb63d3b4a9aecc0d0fb87dd20023ed1a6bd1a9ab44fb1db91580a/black-18.3a2.tar.gz#sha256=2343980b35f425bfb33eebc6bddc4af57ef271c503d30de89f7812c3429a2bc6 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a2
  Found link https://files.pythonhosted.org/packages/3a/ae/3e4c6b429af58b100d3cbdd1b7c0ba0d3d8b74781761961b4ede32390810/black-18.3a3-py36-none-any.whl#sha256=4d21d41c23b8bf4b063ca57353414cba77f554d3360a7d538954c1de5b8df664 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a3
  Found link https://files.pythonhosted.org/packages/0f/69/f60a77390ae2182e8370fe669632812915599163853e55d04dec3ebed605/black-18.3a3.tar.gz#sha256=ee8f84ff1e06b5c262e3bad000eaa2a12b0071a4b2f269502114aeaefbdfb198 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a3
  Found link https://files.pythonhosted.org/packages/5e/c7/a1acc4f8f6baecacefabcf79e2165255f5605c7e33108aa58f16f4e55760/black-18.3a4-py36-none-any.whl#sha256=40c7d13615f9f183b5bab52afa4b3fa9eac62549149ba698185a5365db8efffc (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a4
  Found link https://files.pythonhosted.org/packages/f1/c2/a336bc9bf176d30d9824e587fbe6f921ee51d7ab5f67f28ad7c2ee52fc8c/black-18.3a4.tar.gz#sha256=814a1f5c9cf8dd8d325a224aa989be622fbaec9325ba13d884d9cf5485b47056 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a4
  Found link https://files.pythonhosted.org/packages/b0/3d/28412b6baff122e70a1e9e8c9fdf4f50b1a17b010006a0844c49a182ce1d/black-18.4a0-py36-none-any.whl#sha256=3d0b42275fc6362f60eeaecf9168a503085490be4fd780dffa48035271dc8908 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a0
  Found link https://files.pythonhosted.org/packages/26/fc/fe16644348a849e4171d311dd4a9a6f0f4e1ff62d5525091e0965b547d8b/black-18.4a0.tar.gz#sha256=390b3b3ec035dc2ca5e9df6303e6049d800d07b89d01d985db44ffca4976ae11 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a0
  Found link https://files.pythonhosted.org/packages/9e/0f/933f8016959805328371a25899ea124590345969aac8c4822306c7452ec8/black-18.4a1-py36-none-any.whl#sha256=375582bcb6b8345d10475a9298b1357e8002996c3b31df4eba42f5b541e60f96 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a1
  Found link https://files.pythonhosted.org/packages/6b/2a/908a33ca35eb6fb5d8a57351fba24727ba3ed30191035557f57fa6d9cb83/black-18.4a1.tar.gz#sha256=e9823469b247ed5e186c21f7e94c02aafd8d0905729a5b3748b555bcc2637e5b (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a1
  Found link https://files.pythonhosted.org/packages/e1/e6/dee648e0b1c7997eafa14e51911eeacbed026b2611605bd34d0f824ab6d7/black-18.4a2-py36-none-any.whl#sha256=d5cb273f9fc36ff89f07b84efe50a73d9177158162917198ed4263cd1c40d585 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a2
  Found link https://files.pythonhosted.org/packages/e5/3d/a2204bc1f2feaf5db0105db0f7956a591552ca9ee36f91dff37b08962866/black-18.4a2.tar.gz#sha256=cecee6dc0250661cf86dced28adddf3b65bb669636638745e488e7c1d718e8be (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a2
  Found link https://files.pythonhosted.org/packages/5e/62/58d43ed2e23d853461498132945b510601f0f2e4c3b0a4835bf3ba9b94d7/black-18.4a3-py36-none-any.whl#sha256=af80d5990a4af4bd8885847fef659d59c248d694ed7d2aa076ecaeabc6320006 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a3
  Found link https://files.pythonhosted.org/packages/d1/b0/9233ecee3573cc6e9c1fc257af3c90e4d80818f141ee16d7da4a397120f7/black-18.4a3.tar.gz#sha256=b45061e041762296b0cbf8f476a3036109571e73ce6f3e2d10e1ccc4d0827d59 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a3
  Found link https://files.pythonhosted.org/packages/db/c8/c587383e6b57a07b9937ac29ff93f845aa3d344aa9242f0eba13163eaa2b/black-18.4a4-py36-none-any.whl#sha256=dfb0e344587510ef662785f4a8991addb4017840459bb6d1d53d344ab0f10300 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a4
  Found link https://files.pythonhosted.org/packages/65/c1/3d5ad1f94bbc4b233dba82d2934f5a3b15a33ddaf08165cd0c27d8cfe34b/black-18.4a4.tar.gz#sha256=58594128efb5d21f359c8680911241985e627eb3505397f3982ffbae914d8202 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a4
  Found link https://files.pythonhosted.org/packages/c6/3d/0a966ead09d4f81f5887e5126c2da22d6a0efed89de4b4e9dc98ed727efc/black-18.5b0-py36-none-any.whl#sha256=5fec0f25486046b9edb97961c946412ced96021247dd1a60ecd9f0567b68b030 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.5b0
  Found link https://files.pythonhosted.org/packages/50/3d/887dcd61f361167438af8002cde23376c79439954bd37cbb732be8cf2ab3/black-18.5b0.tar.gz#sha256=4fec2566f9fbbd4a58de50a168cbe3ab952713530410d227e82e4c65d1fad946 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.5b0
  Found link https://files.pythonhosted.org/packages/27/f8/736d91bc142221f0505008059f2eb621e4a753a1236183dc96c0d81b52da/black-18.5b1-py36-none-any.whl#sha256=fa8d4b58aeb454443d4ef02c9daa615cd9a241f0cbe5f2b38b5965209d26dd1c (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.5b1
  Found link https://files.pythonhosted.org/packages/e5/0b/2b5abd975559a5e4cb5cf9697ca242878b4894dc731dd133e582ce459869/black-18.5b1.tar.gz#sha256=c956857a831e508e75f0dc754b86982bad98d95c96aae39c20749209b3958ad7 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.5b1
  Found link https://files.pythonhosted.org/packages/fe/48/d20ab5b038ac70d834b512db343c8fd4d172b35cd541e9544ea3501e3fa6/black-18.6b0-py36-none-any.whl#sha256=8a1220b97ee524e32dcab0e5703e82af96f73b91df86ca0d634fbc4cddde9b25 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b0
  Found link https://files.pythonhosted.org/packages/21/c3/b37ce7f074c6d10ff1fad31edce423502bf50830e74b7a93a2c0ace0d1a2/black-18.6b0.tar.gz#sha256=0e4a4e8b2cae2797b90e300b3e8fa2011557e2f697367ea8a85164b60a37a812 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b0
  Found link https://files.pythonhosted.org/packages/0e/8a/b902cfa966631ee02ecf76ca4466aee800766100a7dd7b2966cf143aa93f/black-18.6b1-py36-none-any.whl#sha256=80dbe2103e937de543942b7946fe71adc387506fe1922018e2b97555513a35fc (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b1
  Found link https://files.pythonhosted.org/packages/70/64/c8257484b57ed564e82170ceced2f8ea8fe57356a0801ffec5d162d19367/black-18.6b1.tar.gz#sha256=0c07b68fc6fc4df8b09873e81893d7b77d52794fa3431d8843b590bc33956105 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b1
  Found link https://files.pythonhosted.org/packages/f5/25/0badb111a8dd98216f55d60c0981d7c64d66e26fde67cc01944da285cb0b/black-18.6b2-py36-none-any.whl#sha256=3efe92eafbde15f8ac06478de11cfb84e47504896ccdde64507e751d2f91ec3a (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b2
  Found link https://files.pythonhosted.org/packages/2c/f1/4e63113f2ee5bd47640c4884ccbed07277d738602ea6d61e0971f527e05a/black-18.6b2.tar.gz#sha256=fc26c4ab28c541fb824f59fa83d5702f75829495d5a1dee603b29bc4fbe79095 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b2
  Found link https://files.pythonhosted.org/packages/c4/53/7a5b0ac836f128a83d837c5e39ee6eb48657b17082f6fe6e21d5fa80871d/black-18.6b3-py36-none-any.whl#sha256=479cc8b3455a75b5b289276b5f47fd2bb412f2f369292989c12b891264758b5c (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b3
  Found link https://files.pythonhosted.org/packages/28/db/fc2361255cea2010fd814edc44c50d118bfcdd031ae7c47ad15b01d3e38d/black-18.6b3.tar.gz#sha256=fe3b7ac846f2a7c91d926782184826c57d2be283c57f0d6b37b85496eb5469ff (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b3
  Found link https://files.pythonhosted.org/packages/4e/b5/575cdaabf1f98ebe50c0ee85b202fcce3b8df1f21d6ed02708bc539674c3/black-18.6b4-py36-none-any.whl#sha256=4b475bbd528acce094c503a3d2dbc2d05a4075f6d0ef7d9e7514518e14cc5191 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b4
  Found link https://files.pythonhosted.org/packages/2d/43/64205493cfb4c8e1720208d73502a121ace2a195cd4d1d49bd7470e4fd92/black-18.6b4.tar.gz#sha256=22158b89c1a6b4eb333a1e65e791a3f8b998cf3b11ae094adb2570f31f769a44 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b4
  Found link https://files.pythonhosted.org/packages/2a/34/9938749f260a861cdd8427d63899e08f9a2a041159a26c2615b02828c973/black-18.9b0-py36-none-any.whl#sha256=817243426042db1d36617910df579a54f1afd659adb96fc5032fcf4b36209739 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.9b0
  Found link https://files.pythonhosted.org/packages/e8/5f/0f79fcd943ba465cbd4bf303c9794970c13a95e5456630de9f72e7f37ad4/black-18.9b0.tar.gz#sha256=e030a9a28f542debc08acceb273f228ac422798e5215ba2a791a6ddeaaca22a5 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.9b0
  Found link https://files.pythonhosted.org/packages/30/62/cf549544a5fe990bbaeca21e9c419501b2de7a701ab0afb377bc81676600/black-19.3b0-py36-none-any.whl#sha256=09a9dcb7c46ed496a9850b76e4e825d6049ecd38b611f1224857a79bd985a8cf (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 19.3b0
  Found link https://files.pythonhosted.org/packages/89/07/aebb10fb8f2ffbac672dfbebffa724643bc84cf012a57737a622d1dabddb/black-19.3b0.tar.gz#sha256=68950ffd4d9169716bcb8719a56c07a2f4485354fec061cdd5910aa07369731c (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 19.3b0
  Found link https://files.pythonhosted.org/packages/fd/bb/ad34bbc93d1bea3de086d7c59e528d4a503ac8fe318bd1fa48605584c3d2/black-19.10b0-py36-none-any.whl#sha256=1b30e59be925fafc1ee4565e5e08abef6b03fe455102883820fe5ee2e4734e0b (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 19.10b0
  Found link https://files.pythonhosted.org/packages/b0/dc/ecd83b973fb7b82c34d828aad621a6e5865764d52375b8ac1d7a45e23c8d/black-19.10b0.tar.gz#sha256=c2edb73a08e9e0e6f65a0e6af18b059b8b1cdd5bef997d7a0b181df93dc81539 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 19.10b0
  Found link https://files.pythonhosted.org/packages/7d/b4/72fb7e851376edf60822318f42143f2984b42f2d69270a115acb6d0ed1e1/black-20.8b0.tar.gz#sha256=0fc6b2f00ccd34cc444d41f24967377c2bc6d0b53f333c56376405ff6cd6f789 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 20.8b0
  Found link https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 20.8b1
Given no hashes to check 42 links for project 'black': discarding no candidates
Using version 20.8b1 (newest of versions: 18.3a0, 18.3a1, 18.3a2, 18.3a3, 18.3a4, 18.4a0, 18.4a1, 18.4a2, 18.4a3, 18.4a4, 18.5b0, 18.5b1, 18.6b0, 18.6b1, 18.6b2, 18.6b3, 18.6b4, 18.9b0, 19.3b0, 19.10b0, 20.8b0, 20.8b1)
Collecting black
  Created temporary directory: /tmp/pip-unpack-imio_gce
  Looking up "https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz" in the cache
  No cache entry available
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz HTTP/1.1" 200 1096433
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 707 kB/s eta 0:00:01  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz"
  Caching due to etag
     |████████████████████████████████| 1.1 MB 707 kB/s
  Saved /tmp/tmpt3ltlh3v/usr.local.bin.python3.7/black-20.8b1.tar.gz
  Added black from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea to build tracker '/tmp/pip-req-tracker-iad7wt7d'
  Created temporary directory: /tmp/pip-build-env-f3a0lj2f
  Running command /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f3a0lj2f/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel
  Non-user install by explicit request
  Created temporary directory: /tmp/pip-ephem-wheel-cache-xosyyour
  Created build tracker: /tmp/pip-req-tracker-iad7wt7d
  Entered build tracker: /tmp/pip-req-tracker-iad7wt7d
  Created temporary directory: /tmp/pip-install-zbaj_ez2
  1 location(s) to search for versions of setuptools:
  * https://pypi.org/simple/setuptools/
  Fetching project page and analyzing links: https://pypi.org/simple/setuptools/
  Getting page https://pypi.org/simple/setuptools/
  Found index url https://pypi.org/simple
  Looking up "https://pypi.org/simple/setuptools/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=4, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (2): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=3, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (3): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (4): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=1, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (5): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=0, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (6): pypi.org:443
  Could not fetch URL https://pypi.org/simple/setuptools/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/setuptools/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))) - skipping
  Given no hashes to check 0 links for project 'setuptools': discarding no candidates
  ERROR: Could not find a version that satisfies the requirement setuptools>=41.0 (from versions: none)
  Cleaning up...
  Removed build tracker: '/tmp/pip-req-tracker-iad7wt7d'
  ERROR: No matching distribution found for setuptools>=41.0
  Exception information:
  Traceback (most recent call last):
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/cli/base_command.py", line 172, in _main
      status = self.run(options, args)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/commands/install.py", line 384, in run
      resolver.resolve(requirement_set)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 185, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 341, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 289, in _get_abstract_dist_for
      req.populate_link(self.finder, upgrade_allowed, require_hashes)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/req/req_install.py", line 252, in populate_link
      self.link = finder.find_requirement(self, upgrade)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/index/package_finder.py", line 925, in find_requirement
      'No matching distribution found for %s' % req
  pip._internal.exceptions.DistributionNotFound: No matching distribution found for setuptools>=41.0
  WARNING: You are using pip version 20.0.dev0; however, version 20.2.3 is available.
  You should consider upgrading via the 'pip install --upgrade pip' command.
  Installing build dependencies ... error
Removed black from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea from build tracker '/tmp/pip-req-tracker-iad7wt7d'
Removed build tracker: '/tmp/pip-req-tracker-iad7wt7d'
ERROR: Command errored out with exit status 1: /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f3a0lj2f/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/cli/base_command.py", line 172, in _main
    status = self.run(options, args)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/commands/download.py", line 146, in run
    resolver.resolve(requirement_set)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 185, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 341, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 290, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/operations/prepare.py", line 544, in prepare_linked_requirement
    req, self.req_tracker, self.finder, self.build_isolation,
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/operations/prepare.py", line 97, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/distributions/source.py", line 38, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/distributions/source.py", line 68, in _setup_isolation
    "Installing build dependencies"
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/build_env.py", line 201, in install_requirements
    call_subprocess(args, spinner=spinner)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f3a0lj2f/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.
pid 64 -> /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5 --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -vvv --cache-dir /root/.pex download --dest /tmp/tmpt3ltlh3v/usr.local.bin.python3.7 --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --cert /ca-cert.crt black setuptools exited with 1 and STDERR:
None
```
Note that black is successfully lokked up and downloaded through the proxy on pypi, but then setuptools fails the same sequence immediately after.


Discussion:
An extra confusing variable is that when REQUESTS_CA_BUNDLE is exported instead of passing --cert, then both black and setuptools are looked up and downloaded successfully through the proxy.
Ok, this is a bug in Pip PEP-517 handling. Notice:
https://github.com/psf/black/blob/235412635e91950c8ef2d9ebe777f97fffd4f01d/pyproject.toml#L36-L38

In other words, black uses pyproject.toml and specifies a custom builder:
```toml
[build-system]
requires = ["setuptools>=41.0", "setuptools-scm", "wheel"]
build-backend = "setuptools.build_meta"
```
Its the execution of the custom builder that fails here in the prerequisite step of downloading its requirements. That seems to point to Pip not propagating its `--cert` setting to subprocesses.
And the Pip issue is here: https://github.com/pypa/pip/issues/5502
OK, and it looks like we can convert --cert into a PIP_CERT env var to solve this: https://github.com/pypa/pip/issues/5502#issuecomment-605658043
Currently we set up `--cert` here in `_calculate_package_index_options`:
https://github.com/pantsbuild/pex/blob/0a29161e3b24a1b7a884ab3acd5f2f411bac7336/pex/pip.py#L173-L175

The assumption in that method is CLI args are returned based on network configuration. We'll probably need to split that method into `_calculate_package_index_flags` and `_calculate_package_index_env` where `_calculate_package_index_env` will yield - for now - this one new env var.


Resolving patch:
diff --git a/pex/pip.py b/pex/pip.py
--- a/pex/pip.py
+++ b/pex/pip.py
@@ -12,12 +12,17 @@
 from pex.common import atomic_directory
 from pex.compatibility import urlparse
 from pex.distribution_target import DistributionTarget
+from pex.interpreter import PythonInterpreter
 from pex.jobs import Job
 from pex.network_configuration import NetworkConfiguration
 from pex.third_party import isolated
 from pex.tracer import TRACER
+from pex.typing import TYPE_CHECKING, cast
 from pex.variables import ENV
 
+if TYPE_CHECKING:
+    from typing import Iterator, List, Optional, Tuple, Mapping
+
 
 class Pip(object):
     @classmethod
@@ -65,7 +70,14 @@ def create(cls, path):
     def __init__(self, pip_pex_path):
         self._pip_pex_path = pip_pex_path
 
-    def _spawn_pip_isolated(self, args, cache=None, interpreter=None):
+    def _spawn_pip_isolated(
+        self,
+        args,  # type: List[str]
+        env=None,  # type: Optional[Mapping[str, str]]
+        cache=None,  # type: Optional[str]
+        interpreter=None,  # type: Optional[PythonInterpreter]
+    ):
+        # type: (...) -> Job
         pip_args = [
             # We vendor the version of pip we want so pip should never check for updates.
             "--disable-pip-version-check",
@@ -83,7 +95,11 @@ def _spawn_pip_isolated(self, args, cache=None, interpreter=None):
         ]
 
         # The max pip verbosity is -vvv and for pex it's -vvvvvvvvv; so we scale down by a factor of 3.
-        pex_verbosity = ENV.PEX_VERBOSE
+
+        # We cast to work around the current awkward state of `Variables` which can only return
+        # `None` in a non-production states.
+        # TODO(John Sirois): https://github.com/pantsbuild/pex/issues/1059
+        pex_verbosity = cast(int, ENV.PEX_VERBOSE)
         pip_verbosity = pex_verbosity // 3
         if pip_verbosity > 0:
             pip_args.append("-{}".format("v" * pip_verbosity))
@@ -96,8 +112,13 @@ def _spawn_pip_isolated(self, args, cache=None, interpreter=None):
             pip_args.append("--no-cache-dir")
 
         command = pip_args + args
+
+        # We cast to work around the current awkward state of `Variables` which can only return
+        # `None` in a non-production states.
+        # TODO(John Sirois): https://github.com/pantsbuild/pex/issues/1059
+        pex_root = cast(str, ENV.PEX_ROOT)
         with ENV.strip().patch(
-            PEX_ROOT=cache or ENV.PEX_ROOT, PEX_VERBOSE=str(pex_verbosity)
+            PEX_ROOT=cache or pex_root, PEX_VERBOSE=str(pex_verbosity), **(env or {})
         ) as env:
             # Guard against API calls from environment with ambient PYTHONPATH preventing pip PEX
             # bootstrapping. See: https://github.com/pantsbuild/pex/issues/892
@@ -114,9 +135,13 @@ def _spawn_pip_isolated(self, args, cache=None, interpreter=None):
                 command=pip.cmdline(command), process=pip.run(args=command, env=env, blocking=False)
             )
 
-    def _calculate_package_index_options(
-        self, indexes=None, find_links=None, network_configuration=None
+    def _calculate_package_index_args(
+        self,
+        indexes=None,  # type: Optional[List[str]]
+        find_links=None,  # type: Optional[List[str]]
+        network_configuration=None,  # type: Optional[NetworkConfiguration]
     ):
+        # type: (...) -> Iterator[str]
         trusted_hosts = []
 
         def maybe_trust_insecure_host(url):
@@ -170,14 +195,21 @@ def maybe_trust_insecure_host(url):
             yield "--proxy"
             yield network_configuration.proxy
 
-        if network_configuration.cert:
-            yield "--cert"
-            yield network_configuration.cert
+        # N.B.: `--cert` is passed via env var to work around:
+        #   https://github.com/pypa/pip/issues/5502
+        # See `_calculate_network_configuration_env`.
 
         if network_configuration.client_cert:
             yield "--client-cert"
             yield network_configuration.client_cert
 
+    def _calculate_network_configuration_env(self, network_configuration=None):
+        # type: (Optional[NetworkConfiguration]) -> Iterator[Tuple[str, str]]
+        network_configuration = network_configuration or NetworkConfiguration.create()
+
+        if network_configuration.cert:
+            yield "PIP_CERT", network_configuration.cert
+
     def spawn_download_distributions(
         self,
         download_dir,
@@ -212,12 +244,16 @@ def spawn_download_distributions(
                 )
 
         download_cmd = ["download", "--dest", download_dir]
-        package_index_options = self._calculate_package_index_options(
-            indexes=indexes,
-            find_links=find_links,
-            network_configuration=network_configuration,
+        download_cmd.extend(
+            self._calculate_package_index_args(
+                indexes=indexes,
+                find_links=find_links,
+                network_configuration=network_configuration,
+            )
+        )
+        env = dict(
+            self._calculate_network_configuration_env(network_configuration=network_configuration)
         )
-        download_cmd.extend(package_index_options)
 
         if target.is_foreign:
             # We're either resolving for a different host / platform or a different interpreter for the
@@ -255,7 +291,7 @@ def spawn_download_distributions(
         download_cmd.extend(requirements)
 
         return self._spawn_pip_isolated(
-            download_cmd, cache=cache, interpreter=target.get_interpreter()
+            download_cmd, env=env, cache=cache, interpreter=target.get_interpreter()
         )
 
     def spawn_build_wheels(
@@ -272,15 +308,19 @@ def spawn_build_wheels(
         wheel_cmd = ["wheel", "--no-deps", "--wheel-dir", wheel_dir]
 
         # If the build is PEP-517 compliant it may need to resolve build requirements.
-        package_index_options = self._calculate_package_index_options(
-            indexes=indexes,
-            find_links=find_links,
-            network_configuration=network_configuration,
+        wheel_cmd.extend(
+            self._calculate_package_index_args(
+                indexes=indexes,
+                find_links=find_links,
+                network_configuration=network_configuration,
+            )
+        )
+        env = dict(
+            self._calculate_network_configuration_env(network_configuration=network_configuration)
         )
-        wheel_cmd.extend(package_index_options)
 
         wheel_cmd.extend(distributions)
-        return self._spawn_pip_isolated(wheel_cmd, cache=cache, interpreter=interpreter)
+        return self._spawn_pip_isolated(wheel_cmd, env=env, cache=cache, interpreter=interpreter)
 
     def spawn_install_wheel(self, wheel, install_dir, compile=False, cache=None, target=None):
 


Test patch: