Repository: pex-tool/pex

Issue:
Pex fails to consistently use https_proxy + --cert
No self-contained repro yet, but a user reports the following:

In a container where the ambient REQUESTS_CA_BUNDLE points to a host-system self-signed cert authority:
```
$ docker run --rm -it -e HTTP_PROXY -e HTTPS_PROXY -v $REQUESTS_CA_BUNDLE:/ca-cert.crt -v $(pwd):/app python:3.7 bash
$ pip install --cert /ca-cert.crt pex
$ pex -vvvvvvvvv --cert /ca-cert.crt black setuptools
...
pex: Building pex :: Resolving distributions (['black', 'setuptools']) :: Resolving for:
pex: Spawning a maximum of 12 parallel jobs to process:r/local/bin/python3.7', PythonIdentity('/usr/local/bin/python', 'cp37', 'cp37m', 'manylinux2014_x86_64', (3, 7, 9))))
  DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.7', PythonIdentity('/usr/local/bin/python', 'cp37', 'cp37m', 'manylinux2014_x86_64', (3, 7, 9))))
pex: Hashing pex: 39.5ms
pex: Isolating pex: 97.5ms
pex:   Extracting pex to /root/.pex/isolated/a8ebd39cfba0316d0416132e488ffc14d28893b5: 97.0ms
pex: Vendoring packaging from VendorSpec(key='packaging', requirement='packaging==19.2', rewrite=True, constrain=True) @ /usr/local/lib/python3.7/site-packages/pex/vendor/_vendored/packaging
pex: Vendoring pyparsing from VendorSpec(key='packaging', requirement='packaging==19.2', rewrite=True, constrain=True) @ /usr/local/lib/python3.7/site-packages/pex/vendor/_vendored/packaging
pex: Vendoring pkg_resources from VendorSpec(key='setuptools', requirement='setuptools==42.0.2', rewrite=True, constrain=True) @ /usr/local/lib/python3.7/site-packages/pex/vendor/_vendored/setuptools
pex: Executing: /usr/local/bin/python3.7 -s -E /tmp/tmplte70n5r
pex: PEX.run invoking /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5 --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -vvv --cache-dir /root/.pex download --dest /tmp/tmpt3ltlh3v/usr.local.bin.python3.7 --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --cert /ca-cert.crt black setuptools
pex: Installed VendorImporter(root='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', importables=(_Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: Installed VendorImporter(root='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', importables=(_Importable(module='compiler', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='distribution_target', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='resolver', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='variables', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='executor', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='version', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_info', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='tracer', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='environment', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='testing', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='compatibility', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='bootstrap', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='network_configuration', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='__main__', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='util', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_bootstrapper', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='requirements', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_builder', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='jobs', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='typing', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pip', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='interpreter_constraints', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='interpreter', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='finders', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='platforms', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='orderedset', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='common', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_warnings', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='dist_metadata', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='vendor', is_pkg=True, path='pex', prefix='_pex'), _Importable(module='third_party', is_pkg=True, path='pex', prefix='_pex')))
pex: pex.third_party.packaging imported via _Loader(module_name='pex.third_party.packaging', vendor_module_name='pex.vendor._vendored.packaging.packaging')
pex: pex.third_party.pyparsing imported via _Loader(module_name='pex.third_party.pyparsing', vendor_module_name='pex.vendor._vendored.packaging.pyparsing')
pex: pex.third_party.pkg_resources imported via _Loader(module_name='pex.third_party.pkg_resources', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources')
pex: pex.third_party.pkg_resources.extern imported via _Loader(module_name='pex.third_party.pkg_resources.extern', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern')
pex: pex.third_party.pkg_resources.extern.six imported via _Loader(module_name='pex.third_party.pkg_resources.extern.six', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.six')
pex: pex.third_party.pkg_resources.extern.six.moves imported via _Loader(module_name='pex.third_party.pkg_resources.extern.six.moves', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.six.moves')
pex: pex.third_party.pkg_resources.extern.packaging imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging')
pex: pex.third_party.pkg_resources.extern.packaging.version imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging.version', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.version')
pex: pex.third_party.pkg_resources.extern.packaging.specifiers imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging.specifiers', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.specifiers')
pex: pex.third_party.pkg_resources.extern.pyparsing imported via _Loader(module_name='pex.third_party.pkg_resources.extern.pyparsing', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.pyparsing')
pex: pex.third_party.pkg_resources.extern.six.moves.urllib imported via _Loader(module_name='pex.third_party.pkg_resources.extern.six.moves.urllib', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.six.moves.urllib')
pex: pex.third_party.pkg_resources.extern.packaging.requirements imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging.requirements', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.requirements')
pex: pex.third_party.pkg_resources.extern.packaging.markers imported via _Loader(module_name='pex.third_party.pkg_resources.extern.packaging.markers', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.markers')
pex: pex.third_party.pkg_resources imported via _Loader(module_name='pex.third_party.pkg_resources', vendor_module_name='pex.vendor._vendored.setuptools.pkg_resources')
pex: pex.third_party.packaging.specifiers imported via _Loader(module_name='pex.third_party.packaging.specifiers', vendor_module_name='pex.vendor._vendored.packaging.packaging.specifiers')
pex: Using the current interpreter /usr/local/bin/python3.7 since no constraints have been specified and PYTHONPATH is not set.
pex: Selecting runtime interpreter: 0.1ms
pex: Found site-library: /usr/local/lib/python3.7/site-packages
pex: Found .pth file: /usr/local/lib/python3.7/site-packages/distutils-precedence.pth
pex: Not a tainted path element: /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap
pex: Not a tainted path element: /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5
pex: Not a tainted path element: /usr/local/lib/python37.zip
pex: Not a tainted path element: /usr/local/lib/python3.7
pex: Not a tainted path element: /usr/local/lib/python3.7/lib-dynload
pex: Tainted path element: /usr/local/lib/python3.7/site-packages
pex: Scrubbing from user site: /root/.local/lib/python3.7/site-packages
pex: Scrubbing from site-packages: /usr/local/lib/python3.7/site-packages
pex: Dropping _distutils_hack
pex: Dropping setuptools
pex: Dropping _distutils_hack.override
pex: Dropping setuptools._distutils
pex: Dropping distutils
pex: Dropping distutils.core
pex: Dropping distutils.debug
pex: Dropping distutils.errors
pex: Dropping distutils.dist
pex: Dropping distutils.fancy_getopt
pex: Dropping distutils.util
pex: Dropping distutils.dep_util
pex: Dropping distutils.spawn
pex: Dropping distutils.log
pex: Dropping distutils.py35compat
pex: Dropping distutils.cmd
pex: Dropping distutils.dir_util
pex: Dropping distutils.file_util
pex: Dropping distutils.archive_util
pex: Dropping distutils.config
pex: Dropping distutils.extension
pex: Dropping setuptools._deprecation_warning
pex: Dropping setuptools.version
pex: Dropping pkg_resources
pex: Dropping pkg_resources.extern
pex: Dropping pkg_resources._vendor
pex: Dropping pkg_resources._vendor.appdirs
pex: Dropping pkg_resources.extern.appdirs
pex: Dropping pkg_resources._vendor.packaging
pex: Dropping pkg_resources._vendor.packaging.__about__
pex: Dropping pkg_resources.extern.packaging
pex: Dropping pkg_resources.extern.packaging.version
pex: Dropping pkg_resources.extern.packaging._structures
pex: Dropping pkg_resources.extern.packaging._typing
pex: Dropping pkg_resources.extern.packaging.specifiers
pex: Dropping pkg_resources.extern.packaging._compat
pex: Dropping pkg_resources.extern.packaging.utils
pex: Dropping pkg_resources.extern.packaging.requirements
pex: Dropping pkg_resources._vendor.pyparsing
pex: Dropping pkg_resources.extern.pyparsing
pex: Dropping pkg_resources.extern.packaging.markers
pex: Dropping setuptools.extension
pex: Dropping setuptools.monkey
pex: Dropping distutils.filelist
pex: Dropping setuptools.dist
pex: Dropping distutils.version
pex: Dropping setuptools.extern
pex: Dropping setuptools._vendor
pex: Dropping setuptools._vendor.packaging
pex: Dropping setuptools._vendor.packaging.__about__
pex: Dropping setuptools.extern.packaging
pex: Dropping setuptools._vendor.ordered_set
pex: Dropping setuptools.extern.ordered_set
pex: Dropping setuptools.windows_support
pex: Dropping setuptools.config
pex: Dropping setuptools.extern.packaging.version
pex: Dropping setuptools.extern.packaging._structures
pex: Dropping setuptools.extern.packaging._typing
pex: Dropping setuptools.extern.packaging.specifiers
pex: Dropping setuptools.extern.packaging._compat
pex: Dropping setuptools.extern.packaging.utils
pex: Dropping setuptools.depends
pex: Dropping setuptools._imp
pex: Dropping setuptools.py34compat
pex: Dropping setuptools.msvc
pex: Dropping distutils.ccompiler
pex: Dropping distutils.sysconfig
pex: Dropping distutils.text_file
pex: New sys.path: ['/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload']
pex: E: tags for 'manylinux2014_x86_64' x PythonInterpreter('/usr/local/bin/python3.7', PythonIdentity('/usr/local/bin/python3.7', 'cp37', 'cp37m', 'manylinux2014_x86_64', (3, 7, 9))) -> frozenset({<cp37-cp37m-manylinux1_x86_64 @ 140329298470704>, <py30-none-manylinux2014_x86_64 @ 140329298487648>, <cp37-none-manylinux2010_x86_64 @ 140329298471584>, <py30-none-any @ 140329298488928>, <cp35-abi3-manylinux2014_x86_64 @ 140329298472544>, <py34-none-linux_x86_64 @ 140329298485888>, <cp36-abi3-linux_x86_64 @ 140329298472464>, <py35-none-any @ 140329298488528>, <py35-none-linux_x86_64 @ 140329298485328>, <py33-none-linux_x86_64 @ 140329298486448>, <py37-none-manylinux1_x86_64 @ 140329298483392>, <py35-none-manylinux2014_x86_64 @ 140329298484752>, <cp36-abi3-manylinux2014_x86_64 @ 140329298471984>, <cp37-none-any @ 140329298488208>, <py32-none-manylinux1_x86_64 @ 140329298486848>, <cp37-none-manylinux2014_x86_64 @ 140329298471424>, <py36-none-manylinux1_x86_64 @ 140329298484512>, <py30-none-linux_x86_64 @ 140329298488128>, <py34-none-manylinux1_x86_64 @ 140329298485728>, <cp37-abi3-manylinux1_x86_64 @ 140329298471184>, <cp36-abi3-manylinux1_x86_64 @ 140329298472304>, <cp35-abi3-manylinux1_x86_64 @ 140329298472864>, <cp37-none-linux_x86_64 @ 140329298471904>, <py32-none-any @ 140329298488768>, <py37-none-manylinux2014_x86_64 @ 140329298483072>, <cp37-cp37m-linux_x86_64 @ 140329298470384>, <cp34-abi3-linux_x86_64 @ 140329298481872>, <py32-none-manylinux2014_x86_64 @ 140329298486528>, <py37-none-any @ 140329298488288>, <cp37-abi3-manylinux2014_x86_64 @ 140329298470864>, <py31-none-linux_x86_64 @ 140329298487568>, <py31-none-any @ 140329298488848>, <py34-none-any @ 140329298488608>, <py31-none-manylinux2014_x86_64 @ 140329298487088>, <py34-none-manylinux2014_x86_64 @ 140329298485408>, <cp37-cp37m-manylinux2010_x86_64 @ 140329298470544>, <py36-none-linux_x86_64 @ 140329298484672>, <cp37-cp37m-manylinux2014_x86_64 @ 140329298389504>, <py30-none-manylinux1_x86_64 @ 140329298487968>, <py31-none-manylinux2010_x86_64 @ 140329298487248>, <py3-none-manylinux2010_x86_64 @ 140329298483792>, <py35-none-manylinux1_x86_64 @ 140329298485072>, <py33-none-manylinux2010_x86_64 @ 140329298486128>, <py36-none-any @ 140329298488448>, <cp33-abi3-manylinux1_x86_64 @ 140329298482272>, <py33-none-any @ 140329298488688>, <cp32-abi3-manylinux2010_x86_64 @ 140329298482672>, <cp37-none-manylinux1_x86_64 @ 140329298471744>, <py36-none-manylinux2014_x86_64 @ 140329298484192>, <py33-none-manylinux2014_x86_64 @ 140329298485968>, <cp34-abi3-manylinux1_x86_64 @ 140329298481712>, <py36-none-manylinux2010_x86_64 @ 140329298484352>, <cp37-abi3-linux_x86_64 @ 140329298471344>, <cp32-abi3-linux_x86_64 @ 140329298482992>, <cp35-abi3-linux_x86_64 @ 140329298481312>, <cp32-abi3-manylinux2014_x86_64 @ 140329298482512>, <cp36-abi3-manylinux2010_x86_64 @ 140329298472144>, <cp35-abi3-manylinux2010_x86_64 @ 140329298472704>, <py37-none-manylinux2010_x86_64 @ 140329298483232>, <py3-none-manylinux1_x86_64 @ 140329298483952>, <cp33-abi3-linux_x86_64 @ 140329298482432>, <cp34-abi3-manylinux2010_x86_64 @ 140329298481552>, <py32-none-manylinux2010_x86_64 @ 140329298486688>, <cp33-abi3-manylinux2014_x86_64 @ 140329298481952>, <py3-none-any @ 140329298488368>, <py34-none-manylinux2010_x86_64 @ 140329298485568>, <py3-none-manylinux2014_x86_64 @ 140329298483632>, <py32-none-linux_x86_64 @ 140329298487008>, <cp37-abi3-manylinux2010_x86_64 @ 140329298471024>, <cp33-abi3-manylinux2010_x86_64 @ 140329298482112>, <py37-none-linux_x86_64 @ 140329298483552>, <py33-none-manylinux1_x86_64 @ 140329298486288>, <cp34-abi3-manylinux2014_x86_64 @ 140329298481392>, <cp32-abi3-manylinux1_x86_64 @ 140329298482832>, <py30-none-manylinux2010_x86_64 @ 140329298487808>, <py3-none-linux_x86_64 @ 140329298484112>, <py35-none-manylinux2010_x86_64 @ 140329298484912>, <py31-none-manylinux1_x86_64 @ 140329298487408>})
pex: Activating PEX virtual environment from /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5: 10.8ms
pex:   Searching dependency cache: /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps: 1.4ms
pex:     Adding setuptools 42.0.2: 0.1ms
pex:     Adding wheel 0.33.6: 0.1ms
pex:     Adding pip 20.0.dev0: 0.1ms
pex:   Resolving pip from [Requirement.parse('pip==20.0.dev0')]: 2.0ms
pex:   Resolving setuptools from [Requirement.parse('setuptools==42.0.2')]: 2.7ms
pex:   Resolving wheel from [Requirement.parse('wheel==0.33.6')]: 1.9ms
pex:   Activating pip 20.0.dev0: 0.2ms
pex:     Adding sitedir: 0.1ms
pex:   Activating setuptools 42.0.2: 0.3ms
pex:     Adding sitedir: 0.2ms
pex:   Activating wheel 0.33.6: 0.3ms
pex:     Adding sitedir: 0.2ms
pex: Bootstrap complete, performing final sys.path modifications...
pex: Not uninstalling VendorImporter(root='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', importables=(_Importable(module='compiler', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='distribution_target', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='resolver', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='variables', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='executor', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='version', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_info', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='tracer', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='environment', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='testing', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='compatibility', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='bootstrap', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='network_configuration', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='__main__', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='util', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_bootstrapper', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='requirements', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_builder', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='jobs', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='typing', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pip', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='interpreter_constraints', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='interpreter', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='finders', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='platforms', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='orderedset', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='common', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='pex_warnings', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='dist_metadata', is_pkg=False, path='pex', prefix='_pex'), _Importable(module='vendor', is_pkg=True, path='pex', prefix='_pex'), _Importable(module='third_party', is_pkg=True, path='pex', prefix='_pex')))
pex: un-imported pex.third_party.packaging
pex: un-imported pex.third_party.packaging.specifiers
pex: un-imported pex.vendor._vendored.packaging.packaging
pex: un-imported pex.vendor._vendored.packaging.packaging.__about__
pex: un-imported pex.vendor._vendored.packaging.packaging._compat
pex: un-imported pex.vendor._vendored.packaging.packaging._structures
pex: un-imported pex.vendor._vendored.packaging.packaging.markers
pex: un-imported pex.vendor._vendored.packaging.packaging.specifiers
pex: un-imported pex.vendor._vendored.packaging.packaging.tags
pex: un-imported pex.vendor._vendored.packaging.packaging.version
pex: un-imported pex.third_party.pyparsing
pex: un-imported pex.vendor._vendored.packaging.pyparsing
pex: un-imported pex.third_party.pkg_resources
pex: un-imported pex.third_party.pkg_resources.extern
pex: un-imported pex.third_party.pkg_resources.extern.packaging
pex: un-imported pex.third_party.pkg_resources.extern.packaging.markers
pex: un-imported pex.third_party.pkg_resources.extern.packaging.requirements
pex: un-imported pex.third_party.pkg_resources.extern.packaging.specifiers
pex: un-imported pex.third_party.pkg_resources.extern.packaging.version
pex: un-imported pex.third_party.pkg_resources.extern.pyparsing
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources._vendor
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources._vendor.packaging.__about__
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.appdirs
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging._compat
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging._structures
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.markers
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.requirements
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.specifiers
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.version
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.extern.pyparsing
pex: un-imported pex.vendor._vendored.setuptools.pkg_resources.py31compat
pex: Uninstalled VendorImporter(root='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap', importables=(_Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: Demoting code from Bootstrap(sys_path_entry='/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap')
pex: un-imported <module 'pex.version' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/version.py'>
pex: un-imported <module 'pex.vendor._vendored.setuptools' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/vendor/_vendored/setuptools/__init__.py'>
pex: un-imported <module 'pex.vendor._vendored.packaging' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/vendor/_vendored/packaging/__init__.py'>
pex: un-imported <module 'pex.vendor._vendored' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/vendor/_vendored/__init__.py'>
pex: un-imported <module 'pex.vendor' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/vendor/__init__.py'>
pex: un-imported <module 'pex.variables' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/variables.py'>
pex: un-imported <module 'pex.util' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/util.py'>
pex: un-imported <module 'pex.typing' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/typing.py'>
pex: un-imported <module 'pex.tracer' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/tracer.py'>
pex: un-imported <module 'pex.third_party' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/third_party/__init__.py'>
pex: un-imported <module 'pex.platforms' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/platforms.py'>
pex: un-imported <module 'pex.pip' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pip.py'>
pex: un-imported <module 'pex.pex_warnings' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex_warnings.py'>
pex: un-imported <module 'pex.pex_info' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex_info.py'>
pex: un-imported <module 'pex.pex_builder' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex_builder.py'>
pex: un-imported <module 'pex.pex_bootstrapper' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex_bootstrapper.py'>
pex: un-imported <module 'pex.pex' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/pex.py'>
pex: un-imported <module 'pex.orderedset' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/orderedset.py'>
pex: un-imported <module 'pex.network_configuration' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/network_configuration.py'>
pex: un-imported <module 'pex.jobs' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/jobs.py'>
pex: un-imported <module 'pex.interpreter_constraints' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/interpreter_constraints.py'>
pex: un-imported <module 'pex.interpreter' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/interpreter.py'>
pex: un-imported <module 'pex.finders' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/finders.py'>
pex: un-imported <module 'pex.executor' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/executor.py'>
pex: un-imported <module 'pex.environment' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/environment.py'>
pex: un-imported <module 'pex.distribution_target' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/distribution_target.py'>
pex: un-imported <module 'pex.dist_metadata' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/dist_metadata.py'>
pex: un-imported <module 'pex.compiler' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/compiler.py'>
pex: un-imported <module 'pex.compatibility' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/compatibility.py'>
pex: un-imported <module 'pex.common' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/common.py'>
pex: un-imported <module 'pex.bootstrap' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/bootstrap.py'>
pex: un-imported <module 'pex' from '/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex/__init__.py'>
pex: Re-imported pex from ['/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap/pex']
pex: PYTHONPATH contains:
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5
pex:   * /usr/local/lib/python37.zip
pex:     /usr/local/lib/python3.7
pex:     /usr/local/lib/python3.7/lib-dynload
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/setuptools
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/wheel
pex:     /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
Created temporary directory: /tmp/pip-req-tracker-iad7wt7d
Initialized build tracking at /tmp/pip-req-tracker-iad7wt7d
Created build tracker: /tmp/pip-req-tracker-iad7wt7d
Entered build tracker: /tmp/pip-req-tracker-iad7wt7d
Created temporary directory: /tmp/pip-download-42q1y0wc
1 location(s) to search for versions of black:
* https://pypi.org/simple/black/
Fetching project page and analyzing links: https://pypi.org/simple/black/
Getting page https://pypi.org/simple/black/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/black/" in the cache
No cache entry available
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/black/ HTTP/1.1" 200 4035
Updating cache with response from "https://pypi.org/simple/black/"
Caching due to etag
  Found link https://files.pythonhosted.org/packages/22/5a/ebe5881cbf07435ea1ad76119ee142f42fb7a333d562c202db015b959ae7/black-18.3a0-py3-none-any.whl#sha256=7183263650ba3071034e90b40a1ea74abccbd32cf525cef6d7914479dbe7f2fb (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a0
  Found link https://files.pythonhosted.org/packages/83/a2/fae3ddb516a52a0f37e6e2ef3bd789b6d093d2c7015a41a41bff30f39897/black-18.3a0.tar.gz#sha256=0461c7a52b5beb378936bf642753dec7a45305c96c6129d540b9c53227121a5a (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a0
  Found link https://files.pythonhosted.org/packages/61/b4/3a39dcf59342f53680c59a5e44c0b172ea5d6470a5aaba8ae3cdbfccc1e7/black-18.3a1-py3-none-any.whl#sha256=04edc87b51b34a7aa093f39bf207e2ebc3612b26f2c79fdf8ff13b2d60622c9a (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a1
  Found link https://files.pythonhosted.org/packages/1b/4d/ff18ef3018083196b85c843387dfa6a1faf376e8f662f74634bad84092d5/black-18.3a1.tar.gz#sha256=c9662827051587b732ecb4994a268f218eda896137b3ea6b8af8510682438b79 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a1
  Found link https://files.pythonhosted.org/packages/83/12/450fb640290d1499ae3922f95190b66b36918656c68376517a8477c6c4c9/black-18.3a2-py36-none-any.whl#sha256=e18daeede445c8d78da14734fa87015ba94ab3d2320792fdf31d95ff5a8eb8eb (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a2
  Found link https://files.pythonhosted.org/packages/f7/2b/359a321cb63d3b4a9aecc0d0fb87dd20023ed1a6bd1a9ab44fb1db91580a/black-18.3a2.tar.gz#sha256=2343980b35f425bfb33eebc6bddc4af57ef271c503d30de89f7812c3429a2bc6 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a2
  Found link https://files.pythonhosted.org/packages/3a/ae/3e4c6b429af58b100d3cbdd1b7c0ba0d3d8b74781761961b4ede32390810/black-18.3a3-py36-none-any.whl#sha256=4d21d41c23b8bf4b063ca57353414cba77f554d3360a7d538954c1de5b8df664 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a3
  Found link https://files.pythonhosted.org/packages/0f/69/f60a77390ae2182e8370fe669632812915599163853e55d04dec3ebed605/black-18.3a3.tar.gz#sha256=ee8f84ff1e06b5c262e3bad000eaa2a12b0071a4b2f269502114aeaefbdfb198 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a3
  Found link https://files.pythonhosted.org/packages/5e/c7/a1acc4f8f6baecacefabcf79e2165255f5605c7e33108aa58f16f4e55760/black-18.3a4-py36-none-any.whl#sha256=40c7d13615f9f183b5bab52afa4b3fa9eac62549149ba698185a5365db8efffc (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a4
  Found link https://files.pythonhosted.org/packages/f1/c2/a336bc9bf176d30d9824e587fbe6f921ee51d7ab5f67f28ad7c2ee52fc8c/black-18.3a4.tar.gz#sha256=814a1f5c9cf8dd8d325a224aa989be622fbaec9325ba13d884d9cf5485b47056 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.3a4
  Found link https://files.pythonhosted.org/packages/b0/3d/28412b6baff122e70a1e9e8c9fdf4f50b1a17b010006a0844c49a182ce1d/black-18.4a0-py36-none-any.whl#sha256=3d0b42275fc6362f60eeaecf9168a503085490be4fd780dffa48035271dc8908 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a0
  Found link https://files.pythonhosted.org/packages/26/fc/fe16644348a849e4171d311dd4a9a6f0f4e1ff62d5525091e0965b547d8b/black-18.4a0.tar.gz#sha256=390b3b3ec035dc2ca5e9df6303e6049d800d07b89d01d985db44ffca4976ae11 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a0
  Found link https://files.pythonhosted.org/packages/9e/0f/933f8016959805328371a25899ea124590345969aac8c4822306c7452ec8/black-18.4a1-py36-none-any.whl#sha256=375582bcb6b8345d10475a9298b1357e8002996c3b31df4eba42f5b541e60f96 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a1
  Found link https://files.pythonhosted.org/packages/6b/2a/908a33ca35eb6fb5d8a57351fba24727ba3ed30191035557f57fa6d9cb83/black-18.4a1.tar.gz#sha256=e9823469b247ed5e186c21f7e94c02aafd8d0905729a5b3748b555bcc2637e5b (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a1
  Found link https://files.pythonhosted.org/packages/e1/e6/dee648e0b1c7997eafa14e51911eeacbed026b2611605bd34d0f824ab6d7/black-18.4a2-py36-none-any.whl#sha256=d5cb273f9fc36ff89f07b84efe50a73d9177158162917198ed4263cd1c40d585 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a2
  Found link https://files.pythonhosted.org/packages/e5/3d/a2204bc1f2feaf5db0105db0f7956a591552ca9ee36f91dff37b08962866/black-18.4a2.tar.gz#sha256=cecee6dc0250661cf86dced28adddf3b65bb669636638745e488e7c1d718e8be (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a2
  Found link https://files.pythonhosted.org/packages/5e/62/58d43ed2e23d853461498132945b510601f0f2e4c3b0a4835bf3ba9b94d7/black-18.4a3-py36-none-any.whl#sha256=af80d5990a4af4bd8885847fef659d59c248d694ed7d2aa076ecaeabc6320006 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a3
  Found link https://files.pythonhosted.org/packages/d1/b0/9233ecee3573cc6e9c1fc257af3c90e4d80818f141ee16d7da4a397120f7/black-18.4a3.tar.gz#sha256=b45061e041762296b0cbf8f476a3036109571e73ce6f3e2d10e1ccc4d0827d59 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a3
  Found link https://files.pythonhosted.org/packages/db/c8/c587383e6b57a07b9937ac29ff93f845aa3d344aa9242f0eba13163eaa2b/black-18.4a4-py36-none-any.whl#sha256=dfb0e344587510ef662785f4a8991addb4017840459bb6d1d53d344ab0f10300 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a4
  Found link https://files.pythonhosted.org/packages/65/c1/3d5ad1f94bbc4b233dba82d2934f5a3b15a33ddaf08165cd0c27d8cfe34b/black-18.4a4.tar.gz#sha256=58594128efb5d21f359c8680911241985e627eb3505397f3982ffbae914d8202 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.4a4
  Found link https://files.pythonhosted.org/packages/c6/3d/0a966ead09d4f81f5887e5126c2da22d6a0efed89de4b4e9dc98ed727efc/black-18.5b0-py36-none-any.whl#sha256=5fec0f25486046b9edb97961c946412ced96021247dd1a60ecd9f0567b68b030 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.5b0
  Found link https://files.pythonhosted.org/packages/50/3d/887dcd61f361167438af8002cde23376c79439954bd37cbb732be8cf2ab3/black-18.5b0.tar.gz#sha256=4fec2566f9fbbd4a58de50a168cbe3ab952713530410d227e82e4c65d1fad946 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.5b0
  Found link https://files.pythonhosted.org/packages/27/f8/736d91bc142221f0505008059f2eb621e4a753a1236183dc96c0d81b52da/black-18.5b1-py36-none-any.whl#sha256=fa8d4b58aeb454443d4ef02c9daa615cd9a241f0cbe5f2b38b5965209d26dd1c (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.5b1
  Found link https://files.pythonhosted.org/packages/e5/0b/2b5abd975559a5e4cb5cf9697ca242878b4894dc731dd133e582ce459869/black-18.5b1.tar.gz#sha256=c956857a831e508e75f0dc754b86982bad98d95c96aae39c20749209b3958ad7 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.5b1
  Found link https://files.pythonhosted.org/packages/fe/48/d20ab5b038ac70d834b512db343c8fd4d172b35cd541e9544ea3501e3fa6/black-18.6b0-py36-none-any.whl#sha256=8a1220b97ee524e32dcab0e5703e82af96f73b91df86ca0d634fbc4cddde9b25 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b0
  Found link https://files.pythonhosted.org/packages/21/c3/b37ce7f074c6d10ff1fad31edce423502bf50830e74b7a93a2c0ace0d1a2/black-18.6b0.tar.gz#sha256=0e4a4e8b2cae2797b90e300b3e8fa2011557e2f697367ea8a85164b60a37a812 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b0
  Found link https://files.pythonhosted.org/packages/0e/8a/b902cfa966631ee02ecf76ca4466aee800766100a7dd7b2966cf143aa93f/black-18.6b1-py36-none-any.whl#sha256=80dbe2103e937de543942b7946fe71adc387506fe1922018e2b97555513a35fc (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b1
  Found link https://files.pythonhosted.org/packages/70/64/c8257484b57ed564e82170ceced2f8ea8fe57356a0801ffec5d162d19367/black-18.6b1.tar.gz#sha256=0c07b68fc6fc4df8b09873e81893d7b77d52794fa3431d8843b590bc33956105 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b1
  Found link https://files.pythonhosted.org/packages/f5/25/0badb111a8dd98216f55d60c0981d7c64d66e26fde67cc01944da285cb0b/black-18.6b2-py36-none-any.whl#sha256=3efe92eafbde15f8ac06478de11cfb84e47504896ccdde64507e751d2f91ec3a (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b2
  Found link https://files.pythonhosted.org/packages/2c/f1/4e63113f2ee5bd47640c4884ccbed07277d738602ea6d61e0971f527e05a/black-18.6b2.tar.gz#sha256=fc26c4ab28c541fb824f59fa83d5702f75829495d5a1dee603b29bc4fbe79095 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b2
  Found link https://files.pythonhosted.org/packages/c4/53/7a5b0ac836f128a83d837c5e39ee6eb48657b17082f6fe6e21d5fa80871d/black-18.6b3-py36-none-any.whl#sha256=479cc8b3455a75b5b289276b5f47fd2bb412f2f369292989c12b891264758b5c (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b3
  Found link https://files.pythonhosted.org/packages/28/db/fc2361255cea2010fd814edc44c50d118bfcdd031ae7c47ad15b01d3e38d/black-18.6b3.tar.gz#sha256=fe3b7ac846f2a7c91d926782184826c57d2be283c57f0d6b37b85496eb5469ff (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b3
  Found link https://files.pythonhosted.org/packages/4e/b5/575cdaabf1f98ebe50c0ee85b202fcce3b8df1f21d6ed02708bc539674c3/black-18.6b4-py36-none-any.whl#sha256=4b475bbd528acce094c503a3d2dbc2d05a4075f6d0ef7d9e7514518e14cc5191 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b4
  Found link https://files.pythonhosted.org/packages/2d/43/64205493cfb4c8e1720208d73502a121ace2a195cd4d1d49bd7470e4fd92/black-18.6b4.tar.gz#sha256=22158b89c1a6b4eb333a1e65e791a3f8b998cf3b11ae094adb2570f31f769a44 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.6b4
  Found link https://files.pythonhosted.org/packages/2a/34/9938749f260a861cdd8427d63899e08f9a2a041159a26c2615b02828c973/black-18.9b0-py36-none-any.whl#sha256=817243426042db1d36617910df579a54f1afd659adb96fc5032fcf4b36209739 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.9b0
  Found link https://files.pythonhosted.org/packages/e8/5f/0f79fcd943ba465cbd4bf303c9794970c13a95e5456630de9f72e7f37ad4/black-18.9b0.tar.gz#sha256=e030a9a28f542debc08acceb273f228ac422798e5215ba2a791a6ddeaaca22a5 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 18.9b0
  Found link https://files.pythonhosted.org/packages/30/62/cf549544a5fe990bbaeca21e9c419501b2de7a701ab0afb377bc81676600/black-19.3b0-py36-none-any.whl#sha256=09a9dcb7c46ed496a9850b76e4e825d6049ecd38b611f1224857a79bd985a8cf (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 19.3b0
  Found link https://files.pythonhosted.org/packages/89/07/aebb10fb8f2ffbac672dfbebffa724643bc84cf012a57737a622d1dabddb/black-19.3b0.tar.gz#sha256=68950ffd4d9169716bcb8719a56c07a2f4485354fec061cdd5910aa07369731c (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 19.3b0
  Found link https://files.pythonhosted.org/packages/fd/bb/ad34bbc93d1bea3de086d7c59e528d4a503ac8fe318bd1fa48605584c3d2/black-19.10b0-py36-none-any.whl#sha256=1b30e59be925fafc1ee4565e5e08abef6b03fe455102883820fe5ee2e4734e0b (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 19.10b0
  Found link https://files.pythonhosted.org/packages/b0/dc/ecd83b973fb7b82c34d828aad621a6e5865764d52375b8ac1d7a45e23c8d/black-19.10b0.tar.gz#sha256=c2edb73a08e9e0e6f65a0e6af18b059b8b1cdd5bef997d7a0b181df93dc81539 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 19.10b0
  Found link https://files.pythonhosted.org/packages/7d/b4/72fb7e851376edf60822318f42143f2984b42f2d69270a115acb6d0ed1e1/black-20.8b0.tar.gz#sha256=0fc6b2f00ccd34cc444d41f24967377c2bc6d0b53f333c56376405ff6cd6f789 (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 20.8b0
  Found link https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea (from https://pypi.org/simple/black/) (requires-python:>=3.6), version: 20.8b1
Given no hashes to check 42 links for project 'black': discarding no candidates
Using version 20.8b1 (newest of versions: 18.3a0, 18.3a1, 18.3a2, 18.3a3, 18.3a4, 18.4a0, 18.4a1, 18.4a2, 18.4a3, 18.4a4, 18.5b0, 18.5b1, 18.6b0, 18.6b1, 18.6b2, 18.6b3, 18.6b4, 18.9b0, 19.3b0, 19.10b0, 20.8b0, 20.8b1)
Collecting black
  Created temporary directory: /tmp/pip-unpack-imio_gce
  Looking up "https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz" in the cache
  No cache entry available
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz HTTP/1.1" 200 1096433
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 707 kB/s eta 0:00:01  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz"
  Caching due to etag
     |████████████████████████████████| 1.1 MB 707 kB/s
  Saved /tmp/tmpt3ltlh3v/usr.local.bin.python3.7/black-20.8b1.tar.gz
  Added black from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea to build tracker '/tmp/pip-req-tracker-iad7wt7d'
  Created temporary directory: /tmp/pip-build-env-f3a0lj2f
  Running command /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f3a0lj2f/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel
  Non-user install by explicit request
  Created temporary directory: /tmp/pip-ephem-wheel-cache-xosyyour
  Created build tracker: /tmp/pip-req-tracker-iad7wt7d
  Entered build tracker: /tmp/pip-req-tracker-iad7wt7d
  Created temporary directory: /tmp/pip-install-zbaj_ez2
  1 location(s) to search for versions of setuptools:
  * https://pypi.org/simple/setuptools/
  Fetching project page and analyzing links: https://pypi.org/simple/setuptools/
  Getting page https://pypi.org/simple/setuptools/
  Found index url https://pypi.org/simple
  Looking up "https://pypi.org/simple/setuptools/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=4, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (2): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=3, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (3): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (4): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=1, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (5): pypi.org:443
  Incremented Retry for (url='/simple/setuptools/'): Retry(total=0, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/setuptools/
  Starting new HTTPS connection (6): pypi.org:443
  Could not fetch URL https://pypi.org/simple/setuptools/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/setuptools/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))) - skipping
  Given no hashes to check 0 links for project 'setuptools': discarding no candidates
  ERROR: Could not find a version that satisfies the requirement setuptools>=41.0 (from versions: none)
  Cleaning up...
  Removed build tracker: '/tmp/pip-req-tracker-iad7wt7d'
  ERROR: No matching distribution found for setuptools>=41.0
  Exception information:
  Traceback (most recent call last):
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/cli/base_command.py", line 172, in _main
      status = self.run(options, args)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/commands/install.py", line 384, in run
      resolver.resolve(requirement_set)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 185, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 341, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 289, in _get_abstract_dist_for
      req.populate_link(self.finder, upgrade_allowed, require_hashes)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/req/req_install.py", line 252, in populate_link
      self.link = finder.find_requirement(self, upgrade)
    File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/index/package_finder.py", line 925, in find_requirement
      'No matching distribution found for %s' % req
  pip._internal.exceptions.DistributionNotFound: No matching distribution found for setuptools>=41.0
  WARNING: You are using pip version 20.0.dev0; however, version 20.2.3 is available.
  You should consider upgrading via the 'pip install --upgrade pip' command.
  Installing build dependencies ... error
Removed black from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea from build tracker '/tmp/pip-req-tracker-iad7wt7d'
Removed build tracker: '/tmp/pip-req-tracker-iad7wt7d'
ERROR: Command errored out with exit status 1: /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f3a0lj2f/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/cli/base_command.py", line 172, in _main
    status = self.run(options, args)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/commands/download.py", line 146, in run
    resolver.resolve(requirement_set)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 185, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 341, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/legacy_resolve.py", line 290, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/operations/prepare.py", line 544, in prepare_linked_requirement
    req, self.req_tracker, self.finder, self.build_isolation,
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/operations/prepare.py", line 97, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/distributions/source.py", line 38, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/distributions/source.py", line 68, in _setup_isolation
    "Installing build dependencies"
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/build_env.py", line 201, in install_requirements
    call_subprocess(args, spinner=spinner)
  File "/root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5/.deps/pip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f3a0lj2f/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.
pid 64 -> /usr/local/bin/python3.7 /root/.pex/pip.pex/a8ebd39cfba0316d0416132e488ffc14d28893b5 --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -vvv --cache-dir /root/.pex download --dest /tmp/tmpt3ltlh3v/usr.local.bin.python3.7 --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --cert /ca-cert.crt black setuptools exited with 1 and STDERR:
None
```
Note that black is successfully lokked up and downloaded through the proxy on pypi, but then setuptools fails the same sequence immediately after.


Discussion:
An extra confusing variable is that when REQUESTS_CA_BUNDLE is exported instead of passing --cert, then both black and setuptools are looked up and downloaded successfully through the proxy.
Ok, this is a bug in Pip PEP-517 handling. Notice:
https://github.com/psf/black/blob/235412635e91950c8ef2d9ebe777f97fffd4f01d/pyproject.toml#L36-L38

In other words, black uses pyproject.toml and specifies a custom builder:
```toml
[build-system]
requires = ["setuptools>=41.0", "setuptools-scm", "wheel"]
build-backend = "setuptools.build_meta"
```
Its the execution of the custom builder that fails here in the prerequisite step of downloading its requirements. That seems to point to Pip not propagating its `--cert` setting to subprocesses.
And the Pip issue is here: https://github.com/pypa/pip/issues/5502
OK, and it looks like we can convert --cert into a PIP_CERT env var to solve this: https://github.com/pypa/pip/issues/5502#issuecomment-605658043
Currently we set up `--cert` here in `_calculate_package_index_options`:
https://github.com/pantsbuild/pex/blob/0a29161e3b24a1b7a884ab3acd5f2f411bac7336/pex/pip.py#L173-L175

The assumption in that method is CLI args are returned based on network configuration. We'll probably need to split that method into `_calculate_package_index_flags` and `_calculate_package_index_env` where `_calculate_package_index_env` will yield - for now - this one new env var.

As noted in #1060 this is still not completely fixed.

Resolving patch:
diff --git a/pex/pip.py b/pex/pip.py
--- a/pex/pip.py
+++ b/pex/pip.py
@@ -17,19 +17,138 @@
 from pex.network_configuration import NetworkConfiguration
 from pex.third_party import isolated
 from pex.tracer import TRACER
-from pex.typing import TYPE_CHECKING, cast
+from pex.typing import TYPE_CHECKING
 from pex.variables import ENV
 
 if TYPE_CHECKING:
-    from typing import Iterator, List, Optional, Tuple, Mapping
+    from typing import Iterable, Iterator, List, Mapping, Optional, Tuple
+
+
+class PackageIndexConfiguration(object):
+    @staticmethod
+    def _calculate_args(
+        indexes=None,  # type: Optional[List[str]]
+        find_links=None,  # type: Optional[List[str]]
+        network_configuration=None,  # type: Optional[NetworkConfiguration]
+    ):
+        # type: (...) -> Iterator[str]
+
+        # N.B.: `--cert` and `--client-cert` are passed via env var to work around:
+        #   https://github.com/pypa/pip/issues/5502
+        # See `_calculate_env`.
+
+        trusted_hosts = []
+
+        def maybe_trust_insecure_host(url):
+            url_info = urlparse.urlparse(url)
+            if "http" == url_info.scheme:
+                # Implicitly trust explicitly asked for http indexes and find_links repos instead of
+                # requiring separate trust configuration.
+                trusted_hosts.append(url_info.netloc)
+            return url
+
+        # N.B.: We interpret None to mean accept pip index defaults, [] to mean turn off all index
+        # use.
+        if indexes is not None:
+            if len(indexes) == 0:
+                yield "--no-index"
+            else:
+                all_indexes = deque(indexes)
+                yield "--index-url"
+                yield maybe_trust_insecure_host(all_indexes.popleft())
+                if all_indexes:
+                    for extra_index in all_indexes:
+                        yield "--extra-index-url"
+                        yield maybe_trust_insecure_host(extra_index)
+
+        if find_links:
+            for find_link_url in find_links:
+                yield "--find-links"
+                yield maybe_trust_insecure_host(find_link_url)
+
+        for trusted_host in trusted_hosts:
+            yield "--trusted-host"
+            yield trusted_host
+
+        network_configuration = network_configuration or NetworkConfiguration.create()
+
+        # N.B.: Pip sends `Cache-Control: max-age=0` by default which turns of HTTP caching as per
+        # the spec:
+        yield "--header"
+        yield "Cache-Control:max-age={}".format(network_configuration.cache_ttl)
+
+        for header in network_configuration.headers:
+            yield "--header"
+            yield header
+
+        yield "--retries"
+        yield str(network_configuration.retries)
+
+        yield "--timeout"
+        yield str(network_configuration.timeout)
+
+        if network_configuration.proxy:
+            yield "--proxy"
+            yield network_configuration.proxy
+
+    @staticmethod
+    def _calculate_env(
+        network_configuration,  # type: NetworkConfiguration
+        isolated,  # type: bool
+    ):
+        # type: (...) -> Iterator[Tuple[str, str]]
+        if network_configuration.cert:
+            # We use the backdoor of requests (which is vendored by Pip to handle all network
+            # operations) support for REQUESTS_CA_BUNDLE when possible to continue to allow Pip to
+            # operate in `--isolated` mode.
+            yield ("REQUESTS_CA_BUNDLE" if isolated else "PIP_CERT"), network_configuration.cert
+
+        if network_configuration.client_cert:
+            assert not isolated
+            yield "PIP_CLIENT_CERT", network_configuration.cert
+
+    @classmethod
+    def create(
+        cls,
+        indexes=None,  # type: Optional[List[str]]
+        find_links=None,  # type: Optional[List[str]]
+        network_configuration=None,  # type: Optional[NetworkConfiguration]
+    ):
+        # type: (...) -> PackageIndexConfiguration
+        network_configuration = network_configuration or NetworkConfiguration.create()
+
+        # We must pass `--client-cert` via PIP_CLIENT_CERT to work around
+        # https://github.com/pypa/pip/issues/5502. We can only do this by breaking Pip `--isolated`
+        # mode.
+        isolated = not network_configuration.client_cert
+
+        return cls(
+            args=cls._calculate_args(
+                indexes=indexes, find_links=find_links, network_configuration=network_configuration
+            ),
+            env=cls._calculate_env(network_configuration=network_configuration, isolated=isolated),
+            isolated=isolated,
+        )
+
+    def __init__(
+        self,
+        args,  # type: Iterable[str]
+        env,  # type: Iterable[Tuple[str, str]]
+        isolated,  # type: bool
+    ):
+        # type: (...) -> None
+        self.args = tuple(args)  # type: Iterable[str]
+        self.env = dict(env)  # type: Mapping[str, str]
+        self.isolated = isolated  # type: bool
 
 
 class Pip(object):
     @classmethod
     def create(cls, path):
+        # type: (str) -> Pip
         """Creates a pip tool with PEX isolation at path.
 
-        :param str path: The path to build the pip tool pex at.
+        :param path: The path to build the pip tool pex at.
         """
         pip_pex_path = os.path.join(path, isolated().pex_hash)
         with atomic_directory(pip_pex_path, exclusive=True) as chroot:
@@ -68,12 +187,13 @@ def create(cls, path):
         return cls(pip_pex_path)
 
     def __init__(self, pip_pex_path):
-        self._pip_pex_path = pip_pex_path
+        # type: (str) -> None
+        self._pip_pex_path = pip_pex_path  # type: str
 
     def _spawn_pip_isolated(
         self,
-        args,  # type: List[str]
-        env=None,  # type: Optional[Mapping[str, str]]
+        args,  # type: Iterable[str]
+        package_index_configuration=None,  # type: Optional[PackageIndexConfiguration]
         cache=None,  # type: Optional[str]
         interpreter=None,  # type: Optional[PythonInterpreter]
     ):
@@ -81,20 +201,24 @@ def _spawn_pip_isolated(
         pip_args = [
             # We vendor the version of pip we want so pip should never check for updates.
             "--disable-pip-version-check",
-            # Don't read pip configuration files like `~/.config/pip/pip.conf`.
-            "--isolated",
             # If we want to warn about a version of python we support, we should do it, not pip.
             "--no-python-version-warning",
-            # If pip encounters a duplicate file path during its operations we don't want it to prompt
-            # and we'd also like to know about this since it should never occur. We leverage the pip
-            # global option:
+            # If pip encounters a duplicate file path during its operations we don't want it to
+            # prompt and we'd also like to know about this since it should never occur. We leverage
+            # the pip global option:
             # --exists-action <action>
-            #   Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
+            #   Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup,
+            #   (a)bort.
             "--exists-action",
             "a",
         ]
+        if not package_index_configuration or package_index_configuration.isolated:
+            # Don't read PIP_ environment variables or pip configuration files like
+            # `~/.config/pip/pip.conf`.
+            pip_args.append("--isolated")
 
-        # The max pip verbosity is -vvv and for pex it's -vvvvvvvvv; so we scale down by a factor of 3.
+        # The max pip verbosity is -vvv and for pex it's -vvvvvvvvv; so we scale down by a factor
+        # of 3.
         pip_verbosity = ENV.PEX_VERBOSE // 3
         if pip_verbosity > 0:
             pip_args.append("-{}".format("v" * pip_verbosity))
@@ -106,10 +230,17 @@ def _spawn_pip_isolated(
         else:
             pip_args.append("--no-cache-dir")
 
-        command = pip_args + args
+        command = pip_args + list(args)
+
+        # N.B.: Package index options in Pep always have the same option names, but they are
+        # registered as subcommand-specific, so we must append them here _after_ the pip subcommand
+        # specified in `args`.
+        if package_index_configuration:
+            command.extend(package_index_configuration.args)
 
+        env = package_index_configuration.env if package_index_configuration else {}
         with ENV.strip().patch(
-            PEX_ROOT=cache or ENV.PEX_ROOT, PEX_VERBOSE=str(ENV.PEX_VERBOSE), **(env or {})
+            PEX_ROOT=cache or ENV.PEX_ROOT, PEX_VERBOSE=str(ENV.PEX_VERBOSE), **env
         ) as env:
             # Guard against API calls from environment with ambient PYTHONPATH preventing pip PEX
             # bootstrapping. See: https://github.com/pantsbuild/pex/issues/892
@@ -126,99 +257,22 @@ def _spawn_pip_isolated(
                 command=pip.cmdline(command), process=pip.run(args=command, env=env, blocking=False)
             )
 
-    def _calculate_package_index_args(
-        self,
-        indexes=None,  # type: Optional[List[str]]
-        find_links=None,  # type: Optional[List[str]]
-        network_configuration=None,  # type: Optional[NetworkConfiguration]
-    ):
-        # type: (...) -> Iterator[str]
-        trusted_hosts = []
-
-        def maybe_trust_insecure_host(url):
-            url_info = urlparse.urlparse(url)
-            if "http" == url_info.scheme:
-                # Implicitly trust explicitly asked for http indexes and find_links repos instead of
-                # requiring separate trust configuration.
-                trusted_hosts.append(url_info.netloc)
-            return url
-
-        # N.B.: We interpret None to mean accept pip index defaults, [] to mean turn off all index use.
-        if indexes is not None:
-            if len(indexes) == 0:
-                yield "--no-index"
-            else:
-                all_indexes = deque(indexes)
-                yield "--index-url"
-                yield maybe_trust_insecure_host(all_indexes.popleft())
-                if all_indexes:
-                    for extra_index in all_indexes:
-                        yield "--extra-index-url"
-                        yield maybe_trust_insecure_host(extra_index)
-
-        if find_links:
-            for find_link_url in find_links:
-                yield "--find-links"
-                yield maybe_trust_insecure_host(find_link_url)
-
-        for trusted_host in trusted_hosts:
-            yield "--trusted-host"
-            yield trusted_host
-
-        network_configuration = network_configuration or NetworkConfiguration.create()
-
-        # N.B.: Pip sends `Cache-Control: max-age=0` by default which turns of HTTP caching as per the
-        # spec:
-        yield "--header"
-        yield "Cache-Control:max-age={}".format(network_configuration.cache_ttl)
-
-        for header in network_configuration.headers:
-            yield "--header"
-            yield header
-
-        yield "--retries"
-        yield str(network_configuration.retries)
-
-        yield "--timeout"
-        yield str(network_configuration.timeout)
-
-        if network_configuration.proxy:
-            yield "--proxy"
-            yield network_configuration.proxy
-
-        # N.B.: `--cert` is passed via env var to work around:
-        #   https://github.com/pypa/pip/issues/5502
-        # See `_calculate_network_configuration_env`.
-
-        if network_configuration.client_cert:
-            yield "--client-cert"
-            yield network_configuration.client_cert
-
-    def _calculate_network_configuration_env(self, network_configuration=None):
-        # type: (Optional[NetworkConfiguration]) -> Iterator[Tuple[str, str]]
-        network_configuration = network_configuration or NetworkConfiguration.create()
-
-        if network_configuration.cert:
-            yield "PIP_CERT", network_configuration.cert
-
     def spawn_download_distributions(
         self,
-        download_dir,
-        requirements=None,
-        requirement_files=None,
-        constraint_files=None,
-        allow_prereleases=False,
-        transitive=True,
-        target=None,
-        indexes=None,
-        find_links=None,
-        network_configuration=None,
-        cache=None,
-        build=True,
-        manylinux=None,
-        use_wheel=True,
+        download_dir,  # type: str
+        requirements=None,  # type: Optional[Iterable[str]]
+        requirement_files=None,  # type: Optional[Iterable[str]]
+        constraint_files=None,  # type: Optional[Iterable[str]]
+        allow_prereleases=False,  # type: bool
+        transitive=True,  # type: bool
+        target=None,  # type: Optional[DistributionTarget]
+        package_index_configuration=None,  # type: Optional[PackageIndexConfiguration]
+        cache=None,  # type: Optional[str]
+        build=True,  # type: bool
+        manylinux=None,  # type: Optional[str]
+        use_wheel=True,  # type: bool
     ):
-
+        # type: (...) -> Job
         target = target or DistributionTarget.current()
 
         platform = target.get_platform()
@@ -235,21 +289,10 @@ def spawn_download_distributions(
                 )
 
         download_cmd = ["download", "--dest", download_dir]
-        download_cmd.extend(
-            self._calculate_package_index_args(
-                indexes=indexes,
-                find_links=find_links,
-                network_configuration=network_configuration,
-            )
-        )
-        env = dict(
-            self._calculate_network_configuration_env(network_configuration=network_configuration)
-        )
-
         if target.is_foreign:
-            # We're either resolving for a different host / platform or a different interpreter for the
-            # current platform that we have no access to; so we need to let pip know and not otherwise
-            # pickup platform info from the interpreter we execute pip with.
+            # We're either resolving for a different host / platform or a different interpreter for
+            # the current platform that we have no access to; so we need to let pip know and not
+            # otherwise pickup platform info from the interpreter we execute pip with.
             if manylinux and platform.platform.startswith("linux"):
                 download_cmd.extend(
                     ["--platform", platform.platform.replace("linux", manylinux, 1)]
@@ -279,42 +322,45 @@ def spawn_download_distributions(
             for constraint_file in constraint_files:
                 download_cmd.extend(["--constraint", constraint_file])
 
-        download_cmd.extend(requirements)
+        if requirements:
+            download_cmd.extend(requirements)
 
         return self._spawn_pip_isolated(
-            download_cmd, env=env, cache=cache, interpreter=target.get_interpreter()
+            download_cmd,
+            package_index_configuration=package_index_configuration,
+            cache=cache,
+            interpreter=target.get_interpreter(),
         )
 
     def spawn_build_wheels(
         self,
-        distributions,
-        wheel_dir,
-        interpreter=None,
-        indexes=None,
-        find_links=None,
-        network_configuration=None,
-        cache=None,
+        distributions,  # type: Iterable[str]
+        wheel_dir,  # type: str
+        interpreter=None,  # type: Optional[PythonInterpreter]
+        package_index_configuration=None,  # type: Optional[PackageIndexConfiguration]
+        cache=None,  # type: Optional[str]
     ):
-
+        # type: (...) -> Job
         wheel_cmd = ["wheel", "--no-deps", "--wheel-dir", wheel_dir]
-
-        # If the build is PEP-517 compliant it may need to resolve build requirements.
-        wheel_cmd.extend(
-            self._calculate_package_index_args(
-                indexes=indexes,
-                find_links=find_links,
-                network_configuration=network_configuration,
-            )
-        )
-        env = dict(
-            self._calculate_network_configuration_env(network_configuration=network_configuration)
-        )
-
         wheel_cmd.extend(distributions)
-        return self._spawn_pip_isolated(wheel_cmd, env=env, cache=cache, interpreter=interpreter)
 
-    def spawn_install_wheel(self, wheel, install_dir, compile=False, cache=None, target=None):
+        return self._spawn_pip_isolated(
+            wheel_cmd,
+            # If the build leverages PEP-518 it will need to resolve build requirements.
+            package_index_configuration=package_index_configuration,
+            cache=cache,
+            interpreter=interpreter,
+        )
 
+    def spawn_install_wheel(
+        self,
+        wheel,  # type: str
+        install_dir,  # type: str
+        compile=False,  # type: bool
+        cache=None,  # type: Optional[str]
+        target=None,  # type: Optional[DistributionTarget]
+    ):
+        # type: (...) -> Job
         target = target or DistributionTarget.current()
 
         install_cmd = [
@@ -335,8 +381,9 @@ def spawn_install_wheel(self, wheel, install_dir, compile=False, cache=None, tar
                     "platform.".format(wheel, interpreter)
                 )
 
-            # We're installing a wheel for a foreign platform. This is just an unpacking operation though;
-            # so we don't actually need to perform it with a target platform compatible interpreter.
+            # We're installing a wheel for a foreign platform. This is just an unpacking operation
+            # though; so we don't actually need to perform it with a target platform compatible
+            # interpreter.
             install_cmd.append("--ignore-requires-python")
 
         install_cmd.append("--compile" if compile else "--no-compile")
@@ -348,6 +395,7 @@ def spawn_install_wheel(self, wheel, install_dir, compile=False, cache=None, tar
 
 
 def get_pip():
+    # type: () -> Pip
     """Returns a lazily instantiated global Pip object that is safe for un-coordinated use."""
     global _PIP
     if _PIP is None:
diff --git a/pex/resolver.py b/pex/resolver.py
--- a/pex/resolver.py
+++ b/pex/resolver.py
@@ -19,7 +19,7 @@
 from pex.jobs import Raise, SpawnedJob, execute_parallel
 from pex.orderedset import OrderedSet
 from pex.pex_info import PexInfo
-from pex.pip import get_pip
+from pex.pip import PackageIndexConfiguration, get_pip
 from pex.platforms import Platform
 from pex.requirements import local_project_from_requirement, local_projects_from_requirement_file
 from pex.third_party.packaging.markers import Marker
@@ -172,9 +172,7 @@ class DownloadRequest(
             "constraint_files",
             "allow_prereleases",
             "transitive",
-            "indexes",
-            "find_links",
-            "network_configuration",
+            "package_index_configuration",
             "cache",
             "build",
             "use_wheel",
@@ -223,9 +221,7 @@ def _spawn_download(self, resolved_dists_dir, target):
             allow_prereleases=self.allow_prereleases,
             transitive=self.transitive,
             target=target,
-            indexes=self.indexes,
-            find_links=self.find_links,
-            network_configuration=self.network_configuration,
+            package_index_configuration=self.package_index_configuration,
             cache=self.cache,
             build=self.build,
             manylinux=self.manylinux,
@@ -476,18 +472,14 @@ def __init__(
         self,
         build_requests,
         install_requests,
-        indexes=None,
-        find_links=None,
-        network_configuration=None,
+        package_index_configuration=None,
         cache=None,
         compile=False,
     ):
 
         self._build_requests = build_requests
         self._install_requests = install_requests
-        self._indexes = indexes
-        self._find_links = find_links
-        self._network_configuration = network_configuration
+        self._package_index_configuration = package_index_configuration
         self._cache = cache
         self._compile = compile
 
@@ -516,9 +508,7 @@ def _spawn_wheel_build(self, built_wheels_dir, build_request):
             distributions=[build_request.source_path],
             wheel_dir=build_result.build_dir,
             cache=self._cache,
-            indexes=self._indexes,
-            find_links=self._find_links,
-            network_configuration=self._network_configuration,
+            package_index_configuration=self._package_index_configuration,
             interpreter=build_request.target.get_interpreter(),
         )
         return SpawnedJob.wait(job=build_job, result=build_result)
@@ -903,6 +893,9 @@ def resolve_multi(
 
     workspace = safe_mkdtemp()
 
+    package_index_configuration = PackageIndexConfiguration.create(
+        indexes=indexes, find_links=find_links, network_configuration=network_configuration
+    )
     build_requests, download_results = _download_internal(
         interpreters=interpreters,
         platforms=platforms,
@@ -911,9 +904,7 @@ def resolve_multi(
         constraint_files=constraint_files,
         allow_prereleases=allow_prereleases,
         transitive=transitive,
-        indexes=indexes,
-        find_links=find_links,
-        network_configuration=network_configuration,
+        package_index_configuration=package_index_configuration,
         cache=cache,
         build=build,
         use_wheel=use_wheel,
@@ -931,9 +922,7 @@ def resolve_multi(
     build_and_install_request = BuildAndInstallRequest(
         build_requests=build_requests,
         install_requests=install_requests,
-        indexes=indexes,
-        find_links=find_links,
-        network_configuration=network_configuration,
+        package_index_configuration=package_index_configuration,
         cache=cache,
         compile=compile,
     )
@@ -954,9 +943,7 @@ def _download_internal(
     transitive=True,
     interpreters=None,
     platforms=None,
-    indexes=None,
-    find_links=None,
-    network_configuration=None,
+    package_index_configuration=None,
     cache=None,
     build=True,
     use_wheel=True,
@@ -998,9 +985,7 @@ def iter_targets():
         constraint_files=constraint_files,
         allow_prereleases=allow_prereleases,
         transitive=transitive,
-        indexes=indexes,
-        find_links=find_links,
-        network_configuration=network_configuration,
+        package_index_configuration=package_index_configuration,
         cache=cache,
         build=build,
         use_wheel=use_wheel,
@@ -1093,6 +1078,9 @@ def download(
     :raises ValueError: If `build=False` and `use_wheel=False`.
     """
 
+    package_index_configuration = PackageIndexConfiguration.create(
+        indexes=indexes, find_links=find_links, network_configuration=network_configuration
+    )
     local_distributions, download_results = _download_internal(
         interpreters=interpreters,
         platforms=platforms,
@@ -1101,9 +1089,7 @@ def download(
         constraint_files=constraint_files,
         allow_prereleases=allow_prereleases,
         transitive=transitive,
-        indexes=indexes,
-        find_links=find_links,
-        network_configuration=network_configuration,
+        package_index_configuration=package_index_configuration,
         cache=cache,
         build=build,
         use_wheel=use_wheel,
@@ -1179,12 +1165,13 @@ def install(
         else:
             build_requests.append(BuildRequest.from_local_distribution(local_distribution))
 
+    package_index_configuration = PackageIndexConfiguration.create(
+        indexes=indexes, find_links=find_links, network_configuration=network_configuration
+    )
     build_and_install_request = BuildAndInstallRequest(
         build_requests=build_requests,
         install_requests=install_requests,
-        indexes=indexes,
-        find_links=find_links,
-        network_configuration=network_configuration,
+        package_index_configuration=package_index_configuration,
         cache=cache,
         compile=compile,
     )


Test patch: