Repository: pex-tool/pex

Issue:
pex 2.1.101 issue: virtualenv is not valid
Looks like there's an issue in pex 2.1.101.

How to reproduce: anywhere you've got docker
```
docker run -it --rm centos:7 /bin/bash -c "yum -y install python3 && python3.6 -m pip install pex==2.1.101 && pex --python python3.6 pyyaml==5.4.1"
```

Output:
```
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: download.cf.centos.org
 * extras: download.cf.centos.org
 * updates: download.cf.centos.org
base                                                                                                                                                                                      | 3.6 kB  00:00:00     
extras                                                                                                                                                                                    | 2.9 kB  00:00:00     
updates                                                                                                                                                                                   | 2.9 kB  00:00:00     
(1/4): base/7/x86_64/group_gz                                                                                                                                                             | 153 kB  00:00:00     
(2/4): extras/7/x86_64/primary_db                                                                                                                                                         | 247 kB  00:00:00     
(3/4): base/7/x86_64/primary_db                                                                                                                                                           | 6.1 MB  00:00:00     
(4/4): updates/7/x86_64/primary_db                                                                                                                                                        |  16 MB  00:00:01     
Resolving Dependencies
--> Running transaction check
---> Package python3.x86_64 0:3.6.8-18.el7 will be installed
--> Processing Dependency: python3-libs(x86-64) = 3.6.8-18.el7 for package: python3-3.6.8-18.el7.x86_64
--> Processing Dependency: python3-setuptools for package: python3-3.6.8-18.el7.x86_64
--> Processing Dependency: python3-pip for package: python3-3.6.8-18.el7.x86_64
--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: python3-3.6.8-18.el7.x86_64
--> Running transaction check
---> Package python3-libs.x86_64 0:3.6.8-18.el7 will be installed
--> Processing Dependency: libtirpc.so.1()(64bit) for package: python3-libs-3.6.8-18.el7.x86_64
---> Package python3-pip.noarch 0:9.0.3-8.el7 will be installed
---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed
--> Running transaction check
---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================================================================================================================
 Package                                                  Arch                                         Version                                               Repository                                     Size
=================================================================================================================================================================================================================
Installing:
 python3                                                  x86_64                                       3.6.8-18.el7                                          updates                                        70 k
Installing for dependencies:
 libtirpc                                                 x86_64                                       0.2.4-0.16.el7                                        base                                           89 k
 python3-libs                                             x86_64                                       3.6.8-18.el7                                          updates                                       6.9 M
 python3-pip                                              noarch                                       9.0.3-8.el7                                           base                                          1.6 M
 python3-setuptools                                       noarch                                       39.2.0-10.el7                                         base                                          629 k

Transaction Summary
=================================================================================================================================================================================================================
Install  1 Package (+4 Dependent packages)

Total download size: 9.3 M
Installed size: 48 M
Downloading packages:
warning: /var/cache/yum/x86_64/7/updates/packages/python3-3.6.8-18.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY                                      ]  0.0 B/s |    0 B  --:--:-- ETA 
Public key for python3-3.6.8-18.el7.x86_64.rpm is not installed
(1/5): python3-3.6.8-18.el7.x86_64.rpm                                                                                                                                                    |  70 kB  00:00:00     
Public key for libtirpc-0.2.4-0.16.el7.x86_64.rpm is not installed
(2/5): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                 |  89 kB  00:00:00     
(3/5): python3-setuptools-39.2.0-10.el7.noarch.rpm                                                                                                                                        | 629 kB  00:00:00     
(4/5): python3-pip-9.0.3-8.el7.noarch.rpm                                                                                                                                                 | 1.6 MB  00:00:00     
(5/5): python3-libs-3.6.8-18.el7.x86_64.rpm                                                                                                                                               | 6.9 MB  00:00:01     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                            6.3 MB/s | 9.3 MB  00:00:01     
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-9.2009.0.el7.centos.x86_64 (@CentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                1/5 
  Installing : python3-setuptools-39.2.0-10.el7.noarch                                                                                                                                                       2/5 
  Installing : python3-pip-9.0.3-8.el7.noarch                                                                                                                                                                3/5 
  Installing : python3-3.6.8-18.el7.x86_64                                                                                                                                                                   4/5 
  Installing : python3-libs-3.6.8-18.el7.x86_64                                                                                                                                                              5/5 
  Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                1/5 
  Verifying  : python3-setuptools-39.2.0-10.el7.noarch                                                                                                                                                       2/5 
  Verifying  : python3-libs-3.6.8-18.el7.x86_64                                                                                                                                                              3/5 
  Verifying  : python3-3.6.8-18.el7.x86_64                                                                                                                                                                   4/5 
  Verifying  : python3-pip-9.0.3-8.el7.noarch                                                                                                                                                                5/5 

Installed:
  python3.x86_64 0:3.6.8-18.el7                                                                                                                                                                                  

Dependency Installed:
  libtirpc.x86_64 0:0.2.4-0.16.el7                 python3-libs.x86_64 0:3.6.8-18.el7                 python3-pip.noarch 0:9.0.3-8.el7                 python3-setuptools.noarch 0:39.2.0-10.el7                

Complete!
WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead.
Collecting pex==2.1.101
  Downloading https://files.pythonhosted.org/packages/77/11/696975448606b1b29ebc877aae43035537cb1ddc2e6a8f85a2ee448ce2e2/pex-2.1.101-py2.py3-none-any.whl (2.7MB)
    100% |################################| 2.7MB 352kB/s 
Installing collected packages: pex
Successfully installed pex-2.1.101
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pex/pep_376.py", line 423, in from_prefix_install
    site_packages = find_site_packages_dir(venv_dir=prefix_dir, interpreter=interpreter)
  File "/usr/local/lib/python3.6/site-packages/pex/venv/virtualenv.py", line 105, in find_site_packages_dir
    venv_dir=venv_dir, site_packages_dir=site_packages_dir
pex.venv.virtualenv.InvalidVirtualenvError: The virtualenv at /root/.pex/installed_wheels/4e0583d24c881e14342eaf4ec5fbc97f934b999a6828693a99157fde912540cc/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl.85f410fb476a45dfaeb71ee64fdd4120 is not valid. The expected site-packages directory at /root/.pex/installed_wheels/4e0583d24c881e14342eaf4ec5fbc97f934b999a6828693a99157fde912540cc/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl.85f410fb476a45dfaeb71ee64fdd4120/lib/python3.6/site-packages does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/pex/bin/pex.py", line 782, in main
    env=env,
  File "/usr/local/lib/python3.6/site-packages/pex/result.py", line 103, in catch
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pex/bin/pex.py", line 802, in do_main
    options=options,
  File "/usr/local/lib/python3.6/site-packages/pex/bin/pex.py", line 701, in build_pex
    preserve_log=resolver_configuration.preserve_log,
  File "/usr/local/lib/python3.6/site-packages/pex/resolver.py", line 949, in resolve
    ignore_errors=ignore_errors, max_parallel_jobs=max_parallel_jobs
  File "/usr/local/lib/python3.6/site-packages/pex/resolver.py", line 718, in install_distributions
    max_jobs=max_parallel_jobs,
  File "/usr/local/lib/python3.6/site-packages/pex/jobs.py", line 541, in execute_parallel
    yield spawn_result.spawned_job.await_result()
  File "/usr/local/lib/python3.6/site-packages/pex/jobs.py", line 220, in await_result
    job.wait()
  File "/usr/local/lib/python3.6/site-packages/pex/jobs.py", line 81, in wait
    self._finalize_job()
  File "/usr/local/lib/python3.6/site-packages/pex/jobs.py", line 139, in _finalize_job
    self._finalizer(self._process.returncode)
  File "/usr/local/lib/python3.6/site-packages/pex/pip/tool.py", line 715, in fixup_install
    interpreter=interpreter,
  File "/usr/local/lib/python3.6/site-packages/pex/pep_376.py", line 427, in from_prefix_install
    "{prefix_dir}: {err}".format(prefix_dir=prefix_dir, err=e)
pex.pep_376.RecordNotFoundError: Could not find a site-packages directory under installation prefix /root/.pex/installed_wheels/4e0583d24c881e14342eaf4ec5fbc97f934b999a6828693a99157fde912540cc/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl.85f410fb476a45dfaeb71ee64fdd4120: The virtualenv at /root/.pex/installed_wheels/4e0583d24c881e14342eaf4ec5fbc97f934b999a6828693a99157fde912540cc/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl.85f410fb476a45dfaeb71ee64fdd4120 is not valid. The expected site-packages directory at /root/.pex/installed_wheels/4e0583d24c881e14342eaf4ec5fbc97f934b999a6828693a99157fde912540cc/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl.85f410fb476a45dfaeb71ee64fdd4120/lib/python3.6/site-packages does not exist.
```

It looks like pyyaml is getting installed, but under lib64 instead of lib where pex is looking based on the output:
```
[root@67f5c04250bf /]# ls /root/.pex/installed_wheels/4e0583d24c881e14342eaf4ec5fbc97f934b999a6828693a99157fde912540cc/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl.ef84b8851b22467dbbcc39c67ed88c0d/      
lib64
```


Discussion:
Hrm, thanks @sfc-gh-pbennes. I've got some learning to do here.
Ok, on CentOS 7:
```
[root@4889b7f3184a /]# python3
Python 3.6.8 (default, Nov 16 2020, 16:55:22) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_scheme_names()
('nt', 'nt_user', 'osx_framework_user', 'posix_home', 'posix_prefix', 'posix_user')
>>> sysconfig.get_path_names()
('stdlib', 'platstdlib', 'purelib', 'platlib', 'include', 'scripts', 'data')
>>> sysconfig.get_path("purelib", "posix_prefix")
'/usr/lib/python3.6/site-packages'
>>> sysconfig.get_path("platlib", "posix_prefix")
'/usr/lib64/python3.6/site-packages'
>>> 
```

On Ubuntu 20.04:
```
root@caaad5bc84bd:/# python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_scheme_names()
('nt', 'nt_user', 'osx_framework_user', 'posix_home', 'posix_prefix', 'posix_user')
>>> sysconfig.get_path_names()
('stdlib', 'platstdlib', 'purelib', 'platlib', 'include', 'scripts', 'data')
>>> sysconfig.get_path("purelib", "posix_prefix")
'/usr/lib/python3.8/site-packages'
>>> sysconfig.get_path("platlib", "posix_prefix")
'/usr/lib/python3.8/site-packages'
>>> 
```

Those accurately reflect where packages get installed. On CentOS, in particular, you find pure-python distributions install to `/usr/lib/python3.6/site-packages` whereas platform-specific distributions like PyYAML install to `/usr/lib64/python3.6/site-packages`. On Ubuntu they install to the same place.

Unfortunately, this gives the wrong answer for PyPy>=2.7<3.8 (Pex supports PyPy>=2.7):
```
$ pypy2.7
Python 2.7.18 (8e99af2f9b8e3a58dd8691378a36ef43c3139ee9, Mar 29 2022, 05:16:07)
[PyPy 7.3.9 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>> import sysconfig
>>>> sysconfig.get_scheme_names()
('nt', 'nt_user', 'os2', 'os2_home', 'osx_framework_user', 'posix_home', 'posix_prefix', 'posix_user', 'pypy', 'pypy_nt')
>>>> sysconfig.get_path_names()
('stdlib', 'platstdlib', 'purelib', 'platlib', 'include', 'scripts', 'data')
>>>> sysconfig.get_path("purelib", "posix_prefix")
'/home/jsirois/.pyenv/versions/pypy2.7-7.3.9/lib/pypy2.7/site-packages'
>>>> sysconfig.get_path("platlib", "posix_prefix")
'/home/jsirois/.pyenv/versions/pypy2.7-7.3.9/lib/pypy2.7/site-packages'
>>>> 
```

That is not the right answer. The right answer in this case is:
```
$ pypy2.7
Python 2.7.18 (8e99af2f9b8e3a58dd8691378a36ef43c3139ee9, Mar 29 2022, 05:16:07)
[PyPy 7.3.9 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>> import site
>>>> site.getsitepackages()
['/home/jsirois/.pyenv/versions/pypy2.7-7.3.9/site-packages']
>>>> 
```

Unfortunately though, this gives the _wrong_ answer on Ubuntu CPython 3.9:
```
root@c9904ab338fe:/# python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import site
>>> site.getsitepackages()
['/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.8/dist-packages']
>>> 
```

I'll need to research more on all this non-uniformity in installation schemes.
Ok, it appears sysconfig _does_ give the right answer for PyPy<3.8 and all other Pythons Pex supports when the default scheme is allowed to flow through; ie. `sysconfig.get_path("purelib")` and `sysconfig.get_path("platlib")` give the answers we're looking for consistently across PyPy/CPython and Arch/Ubuntu/CentOS. We'll let CI confirm the MacOS case.
Ok, this does not, in fact, work for at least some Mac system interpreters :/.

Here is the relevant bit of the interpreter info. Not that the `"prefix"` (`sys.prefix`) is not a prefix of `"purelib"` or `"platlib"`:
```json
{
  "abi_tag": "cp38",
  "base_prefix": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8",
  "binary": "/usr/bin/python3",
  "configured_macosx_deployment_target": "10.14.6",
  "env_markers": {
    "implementation_name": "cpython",
    "implementation_version": "3.8.2",
    "os_name": "posix",
    "platform_machine": "x86_64",
    "platform_python_implementation": "CPython",
    "platform_release": "19.6.0",
    "platform_system": "Darwin",
    "platform_version": "Darwin Kernel Version 19.6.0: Tue Jun 21 21:18:39 PDT 2022; root:xnu-6153.141.66~1/RELEASE_X86_64",
    "python_full_version": "3.8.2",
    "python_version": "3.8",
    "sys_platform": "darwin"
  },
  "platform_tag": "macosx_10_15_x86_64",
  "platlib": "/Library/Python/3.8/site-packages",
  "prefix": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8",
  "purelib": "/Library/Python/3.8/site-packages",
  "python_tag": "cp38",
}
```

No clue what to do about this yet.
The struggle is real 😹 
"Python currently supports seven schemes." and proceeds to list 5: (https://docs.python.org/3.8/library/sysconfig.html#installation-paths). There are, in-fact, 7:
```json
{
  "nt": {
    "stdlib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Lib",
    "platstdlib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Lib",
    "purelib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Lib/site-packages",
    "platlib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Lib/site-packages",
    "include": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Include",
    "platinclude": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Include",
    "scripts": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Scripts",
    "data": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8"
  },
  "nt_user": {
    "stdlib": "/Users/runner/Library/Python/3.8/Python38",
    "platstdlib": "/Users/runner/Library/Python/3.8/Python38",
    "purelib": "/Users/runner/Library/Python/3.8/Python38/site-packages",
    "platlib": "/Users/runner/Library/Python/3.8/Python38/site-packages",
    "include": "/Users/runner/Library/Python/3.8/Python38/Include",
    "scripts": "/Users/runner/Library/Python/3.8/Python38/Scripts",
    "data": "/Users/runner/Library/Python/3.8"
  },
  "osx_framework_library": {
    "stdlib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8",
    "platstdlib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8",
    "purelib": "/Library/Python/3.8/site-packages",
    "platlib": "/Library/Python/3.8/site-packages",
    "include": "/Library/Python/3.8/include",
    "platinclude": "/Library/Python/3.8/include",
    "scripts": "/usr/local/bin",
    "data": "/Library/Python/3.8"
  },
  "osx_framework_user": {
    "stdlib": "/Users/runner/Library/Python/3.8/lib/python",
    "platstdlib": "/Users/runner/Library/Python/3.8/lib/python",
    "purelib": "/Users/runner/Library/Python/3.8/lib/python/site-packages",
    "platlib": "/Users/runner/Library/Python/3.8/lib/python/site-packages",
    "include": "/Users/runner/Library/Python/3.8/include",
    "scripts": "/Users/runner/Library/Python/3.8/bin",
    "data": "/Users/runner/Library/Python/3.8"
  },
  "posix_home": {
    "stdlib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python",
    "platstdlib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python",
    "purelib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python",
    "platlib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python",
    "include": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python",
    "platinclude": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python",
    "scripts": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin",
    "data": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8"
  },
  "posix_prefix": {
    "stdlib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8",
    "platstdlib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8",
    "purelib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages",
    "platlib": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages",
    "include": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8",
    "platinclude": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8",
    "scripts": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin",
    "data": "/Applications/Xcode_12.4.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8"
  },
  "posix_user": {
    "stdlib": "/Users/runner/Library/Python/3.8/lib/python3.8",
    "platstdlib": "/Users/runner/Library/Python/3.8/lib/python3.8",
    "purelib": "/Users/runner/Library/Python/3.8/lib/python3.8/site-packages",
    "platlib": "/Users/runner/Library/Python/3.8/lib/python3.8/site-packages",
    "include": "/Users/runner/Library/Python/3.8/include/python3.8",
    "scripts": "/Users/runner/Library/Python/3.8/bin",
    "data": "/Users/runner/Library/Python/3.8"
  }
}
```
At least for MacOS Python 3.8 framework interpreters, the `"osx_framework_library"` scheme appears to be the preferred scheme. The one that would work here in `"posix_prefix"`. I'll need to dig back in to the issues this posed for PyPy and see what I can come up with.
Alright, I think the best path forward is to revert back to the prior naive approach, but instead of simply walking the prefix directory for the 1st 'site-packages' subdir, just collect them all. The current code handles then checking each found `site-packages` for the install and so this should both continue to solve #1861 and robustly find whatever-the-scheme `site-packes` (assuming its called `site-packages`!).

Resolving patch:
diff --git a/pex/pep_376.py b/pex/pep_376.py
--- a/pex/pep_376.py
+++ b/pex/pep_376.py
@@ -20,7 +20,7 @@
 from pex.dist_metadata import Distribution, EntryPoint
 from pex.interpreter import PythonInterpreter
 from pex.typing import TYPE_CHECKING, cast
-from pex.venv.virtualenv import InvalidVirtualenvError, Virtualenv, find_site_packages_dir
+from pex.venv.virtualenv import Virtualenv
 
 if TYPE_CHECKING:
     from typing import (
@@ -29,6 +29,7 @@
         Dict,
         Iterable,
         Iterator,
+        List,
         Optional,
         Protocol,
         Tuple,
@@ -411,31 +412,49 @@ def read(
             size = int(file_size) if file_size else None
             yield InstalledFile(path=path, hash=file_hash, size=size)
 
+    @staticmethod
+    def _find_installation(
+        prefix_dir,  # type: str
+        project_name,  # type: str
+        version,  # type: str
+    ):
+        # type: (...) -> Optional[Tuple[str, str, List[str]]]
+
+        # Some distributions in the wild (namely python-certifi-win32 1.6.1,
+        # see: https://github.com/pantsbuild/pex/issues/1861) create their own directories named
+        # `site-packages` that are not in-fact located in site-packages (the "purelib" or "platlib"
+        # sysconfig install paths). Work around these broken packages by just looking for all
+        # `site-packages` subdirectories of the `prefix_dir` and checking each for the installation
+        # `RECORD`. There should always be just one such installation `RECORD` resulting from a
+        # `pip install --prefix <prefix_dir> --no-deps <wheel file>` and so this is safe.
+        site_packages_dirs = [
+            os.path.join(root, d)
+            for root, dirs, _ in os.walk(prefix_dir)
+            for d in dirs
+            if d == "site-packages"
+        ]
+        for site_packages_dir in site_packages_dirs:
+            site_packages_listing = [
+                os.path.relpath(os.path.join(root, f), site_packages_dir)
+                for root, _, files in os.walk(site_packages_dir)
+                for f in files
+            ]
+            record_relative_path = dist_metadata.find_dist_info_file(
+                project_name, version=version, filename="RECORD", listing=site_packages_listing
+            )
+            if record_relative_path:
+                return record_relative_path, site_packages_dir, site_packages_listing
+        return None
+
     @classmethod
     def from_prefix_install(
         cls,
         prefix_dir,  # type: str
         project_name,  # type: str
         version,  # type: str
-        interpreter=None,  # type: Optional[PythonInterpreter]
     ):
-        try:
-            site_packages = find_site_packages_dir(venv_dir=prefix_dir, interpreter=interpreter)
-        except InvalidVirtualenvError as e:
-            raise RecordNotFoundError(
-                "Could not find a site-packages directory under installation prefix "
-                "{prefix_dir}: {err}".format(prefix_dir=prefix_dir, err=e)
-            )
-
-        site_packages_listing = [
-            os.path.relpath(os.path.join(root, f), site_packages)
-            for root, _, files in os.walk(site_packages)
-            for f in files
-        ]
-        record_relative_path = dist_metadata.find_dist_info_file(
-            project_name, version=version, filename="RECORD", listing=site_packages_listing
-        )
-        if not record_relative_path:
+        result = cls._find_installation(prefix_dir, project_name, version)
+        if not result:
             raise RecordNotFoundError(
                 "Could not find the installation RECORD for {project_name} {version} under "
                 "{prefix_dir}".format(
@@ -443,6 +462,7 @@ def from_prefix_install(
                 )
             )
 
+        record_relative_path, site_packages, site_packages_listing = result
         metadata_dir = os.path.dirname(record_relative_path)
         base_dir = os.path.relpath(site_packages, prefix_dir)
         return cls(
diff --git a/pex/pip/tool.py b/pex/pip/tool.py
--- a/pex/pip/tool.py
+++ b/pex/pip/tool.py
@@ -712,7 +712,6 @@ def fixup_install(returncode):
                 prefix_dir=install_dir,
                 project_name=project_name_and_version.project_name,
                 version=project_name_and_version.version,
-                interpreter=interpreter,
             )
             record.fixup_install(interpreter=interpreter)
 


Test patch: