Repository: pipxproject/pipx (python)

Issue:
ModuleNotFoundError: No module named 'pip._vendor.six' when running "pipx install"
**Describe the bug**

I have installed a new Debian (unstable) system. On one of my installations pipx runs without problems. The newly installed system behaves differently and I do not know where to look.

**How to reproduce**

```
$ pipx install --verbose youtube-dl
pipx > (_package_name_from_spec:93): Determined package name: youtube-dl
pipx > (_package_name_from_spec:94): Package name determined in 0.0s
pipx > (run_subprocess:112): running /usr/bin/python3 -m venv --without-pip /home/kai/.local/pipx/venvs/youtube-dl
pipx > (run_subprocess:112): running /home/kai/.local/pipx/venvs/youtube-dl/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/kai/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/kai/.local/pipx/venvs/youtube-dl/bin/python --version
pipx > (run_subprocess:112): running /home/kai/.local/pipx/venvs/youtube-dl/bin/python -m pip install youtube-dl
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/__main__.py", line 16, in <module>
    from pip._internal.cli.main import main as _main  # isort:skip # noqa
  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 10, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/_internal/cli/cmdoptions.py", line 24, in <module>
    from pip._internal.exceptions import CommandError
  File "/home/kai/.local/pipx/shared/lib/python3.8/site-packages/pip/_internal/exceptions.py", line 10, in <module>
    from pip._vendor.six import iteritems
ModuleNotFoundError: No module named 'pip._vendor.six'
pipx > (install_package:189): '/home/kai/.local/pipx/venvs/youtube-dl/bin/python -m pip install youtube-dl' failed

pipx > (rmdir:18): removing directory /home/kai/.local/pipx/venvs/youtube-dl
Error installing youtube-dl.
```

If I run python3 from the command line I can import the module:

```
$ python3 -c "import pip._vendor.six"
$
```

I have no idea what could be different. I already checked the list of python3-* packages on both systems but that seems to be the same. Any idea what to look for or how to debug the issue?

**Expected behavior**

The package should be installed.


Pull request:
Avoid Debian-patched pip in the shared venv
Should fix #386.

This fix should be more lightweight than #387, and it is simpler than #388.  It tries to find an acceptable middle ground between explicitly supporting Debian's changes to their pip and causing pain to users in order to make a statement.

This patch needs testing.  I haven't come up with a test plan yet (I don't want to mess up my currently-working actual pipx setup), but I'll try to do something with docker.


Patch:
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index e9e0573..81d67de 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,4 +1,11 @@
-<!---
-Thank you for your soon-to-be pull request. Before you submit this, please
-double check to make sure that you've added an entry to docs/changelog.md.
--->
\ No newline at end of file
+<!-- add an 'x' in the brackets below -->
+[] I have added an entry to `docs/changelog.md`
+
+## Summary of changes
+
+## Test plan
+<!-- provide evidence of testing, preferably with command(s) that can be copy+pasted by others -->
+Tested by running
+```
+# command(s) to exercise these changes
+```
diff --git a/.github/workflows/on-push.yml b/.github/workflows/on-push.yml
new file mode 100644
index 0000000..1519cb1
--- /dev/null
+++ b/.github/workflows/on-push.yml
@@ -0,0 +1,84 @@
+# https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: Test
+
+on:
+    pull_request:
+    push:
+    release:
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.8]
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install native-venv version of nox
+      run: |
+        python -m pip install --upgrade pip
+        pip install git+https://github.com/cs01/nox.git@5ea70723e9e6#egg=nox
+    - name: Lint
+      run: |
+        nox --non-interactive --session lint
+
+  docs:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.8]
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install native-venv version of nox
+      run: |
+        python -m pip install --upgrade pip
+        pip install git+https://github.com/cs01/nox.git@5ea70723e9e6#egg=nox
+    - name: Verify Docs
+      run: |
+        nox --non-interactive --session docs
+
+  tests:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest]
+        python-version: [3.6, 3.7, 3.8]
+        include:
+          - os: windows-latest
+            python-version: 3.8
+          - os: macos-latest
+            python-version: 3.8
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Upgrade pip
+      run: python -m pip install --upgrade pip
+    # macos: Install wheel to allow wheel-building for nox and regex
+    # Build regex wheel into pip cache, so future pipx installs don't compile.
+    # This avoids future compile errors in nox tests because of missing
+    #   header files.
+    - name: Install wheel and build wheel for regex to pip cache
+      if: ${{ matrix.os == 'macos-latest' }}
+      run: |
+        python -m pip install wheel
+        python -m pip install regex
+    - name: Install native-venv version of nox
+      run: pip install git+https://github.com/cs01/nox.git@5ea70723e9e6#egg=nox
+    - name: Execute Tests
+      run: |
+        nox --non-interactive --session tests-${{ matrix.python-version }}
diff --git a/.travis.yml b/.travis.yml
deleted file mode 100644
index af06b11..0000000
--- a/.travis.yml
+++ /dev/null
@@ -1,31 +0,0 @@
-language: python
-os: linux
-cache:
-  pip: true
-matrix:
-  include:
-    - python: "3.6"
-      env: NOXSESSION="tests-3.6"
-    - python: "3.7"
-      env: NOXSESSION="tests-3.7"
-      dist: xenial
-    - python: "3.8"
-      env: NOXSESSION="tests-3.8"
-    - python: "3.7"
-      env: NOXSESSION="lint"
-      dist: xenial
-    - python: "3.7"
-      env: NOXSESSION="docs"
-      dist: xenial
-    - os: windows
-      language: bash
-      env: NOXSESSION="tests-3.7"
-      before_install:
-        - export TRAVIS_PYTHON_VERSION=3.7
-        - choco install python --version $TRAVIS_PYTHON_VERSION
-        - export PATH="/C/Python37:/C/Python37/Scripts:$PATH"
-install:
-  - pip install git+https://github.com/cs01/nox.git@5ea70723e9e6#egg=nox
-
-script:
-  - nox --non-interactive --session "$NOXSESSION"
diff --git a/README.md b/README.md
index bb89692..76cbca2 100644
--- a/README.md
+++ b/README.md
@@ -13,8 +13,8 @@
 </p>
 
 <p align="center">
-<a href="https://travis-ci.org/pipxproject/pipx"><img src="https://travis-ci.org/pipxproject/pipx.svg?branch=master" /></a>
-<a href="https://badge.fury.io/py/pipx"><img src="https://badge.fury.io/py/pipx.svg" alt="PyPI version" height="18"></a>
+<a href="https://github.com/pipxproject/pipx/actions?query=workflow%3ATest"><img src="https://github.com/pipxproject/pipx/workflows/Test/badge.svg?branch=master" alt="Test CI" ></a>
+<a href="https://badge.fury.io/py/pipx"><img src="https://badge.fury.io/py/pipx.svg" alt="PyPI version"></a>
 </p>
 
 **Documentation**: https://pipxproject.github.io/pipx/
diff --git a/docs/changelog.md b/docs/changelog.md
index 7c81d2f..4a79c0a 100644
--- a/docs/changelog.md
+++ b/docs/changelog.md
@@ -1,5 +1,7 @@
 dev
+- Change metadata recorded from version-specified install to allow upgrades in future.  Adds pipx dependency on `packaging` package.
 - [bugfix] Prevent python error in case where package has no pipx metadata and advise user how to fix.
+- [feature] `ensurepath` now also ensures that pip user binary path containing pipx itself is in user's PATH if pipx was installed using `pip install --user`.
 - [bugfix] For `pipx install`, fixed failure to install if user has `PIP_USER=1` or `user=true` in pip.conf. (#110)
 - [bugfix] Requiring userpath v1.4.1 or later so ensure Windows bug is fixed for `ensurepath` (#437)
 - [feature] log pipx version (#423)
@@ -26,6 +28,7 @@ dev
 - [bugfix] Hide cursor while pipx runs
 - [feature] Add environment variable `USE_EMOJI` to allow enabling/disabling emojies (#376)
 - [refactor] Moved all commands to separate files within the commands module (#255).
+- [bugfix] Ignore system shared libraries when installing shared libraries pip, wheel, and setuptools. This also fixes an incompatibility with Debian/Ubuntu's version of pip (#386).
 
 0.15.1.3
 
diff --git a/makefile b/makefile
index bfad950..d952144 100644
--- a/makefile
+++ b/makefile
@@ -1,8 +1,11 @@
-.PHONY: test docs develop build publish publish_docs
+.PHONY: test docs develop build publish publish_docs lint
 
 develop:
 	pipx run --spec=git+https://github.com/cs01/nox.git@5ea70723e9e6 nox -s develop
 
+lint:
+	pipx run nox -s lint
+
 test:
 	# TODO use `pipx run nox` when nox supports venv creation (and thus
 	# pipx tests pass)
diff --git a/noxfile.py b/noxfile.py
index a7d0f58..9a168e0 100644
--- a/noxfile.py
+++ b/noxfile.py
@@ -31,7 +31,14 @@ else:
 nox.options.reuse_existing_virtualenvs = True
 
 doc_dependencies = [".", "jinja2", "mkdocs", "mkdocs-material"]
-lint_dependencies = ["black", "flake8", "flake8-bugbear", "mypy", "check-manifest"]
+lint_dependencies = [
+    "black",
+    "flake8",
+    "flake8-bugbear",
+    "mypy",
+    "check-manifest",
+    "packaging>=20.0",
+]
 
 
 @nox.session(python=python)
diff --git a/setup.py b/setup.py
index 29f09df..a138974 100755
--- a/setup.py
+++ b/setup.py
@@ -18,7 +18,11 @@ import re  # noqa E402
 
 CURDIR = Path(__file__).parent
 
-REQUIRED = ["userpath>=1.4.1", "argcomplete>=1.9.4, <2.0"]  # type: List[str]
+REQUIRED = [
+    "userpath>=1.4.1",
+    "argcomplete>=1.9.4, <2.0",
+    "packaging>=20.0",
+]  # type: List[str]
 
 
 def get_version():
diff --git a/src/pipx/commands/__init__.py b/src/pipx/commands/__init__.py
index ca78f3c..bfa3fa6 100644
--- a/src/pipx/commands/__init__.py
+++ b/src/pipx/commands/__init__.py
@@ -1,4 +1,4 @@
-from .ensure_path import ensure_path
+from .ensure_path import ensure_pipx_paths
 from .inject import inject
 from .install import install
 from .list_packages import list_packages
@@ -20,5 +20,5 @@ __all__ = [
     "reinstall_all",
     "list_packages",
     "run_pip",
-    "ensure_path",
+    "ensure_pipx_paths",
 ]
diff --git a/src/pipx/commands/common.py b/src/pipx/commands/common.py
index b040011..be171a4 100644
--- a/src/pipx/commands/common.py
+++ b/src/pipx/commands/common.py
@@ -4,17 +4,18 @@ import shutil
 import sys
 import tempfile
 import time
-import userpath  # type: ignore
-
 from pathlib import Path
 from shutil import which
 from tempfile import TemporaryDirectory
 from typing import Collection, Dict, List, Optional
 
+import userpath  # type: ignore
+
 from pipx import constants
 from pipx.colors import bold, red
 from pipx.emojies import hazard, stars
-from pipx.util import WINDOWS, PipxError, mkdir, rmdir, valid_pypi_name
+from pipx.package_specifier import valid_pypi_name
+from pipx.util import WINDOWS, PipxError, mkdir, rmdir
 from pipx.venv import Venv
 
 
@@ -243,8 +244,11 @@ def package_name_from_spec(
     start_time = time.time()
 
     # shortcut if valid PyPI name and not a local path
-    if valid_pypi_name(package_spec) and not Path(package_spec).exists():
-        package_name = package_spec
+    pypi_name = valid_pypi_name(package_spec)
+    if pypi_name is not None and not Path(package_spec).exists():
+        # NOTE: if pypi name and installed package name differ, this means pipx
+        #       will use the pypi name
+        package_name = pypi_name
         logging.info(f"Determined package name: {package_name}")
         logging.info(f"Package name determined in {time.time()-start_time:.1f}s")
         return package_name
diff --git a/src/pipx/commands/ensure_path.py b/src/pipx/commands/ensure_path.py
index c270201..403a9ff 100644
--- a/src/pipx/commands/ensure_path.py
+++ b/src/pipx/commands/ensure_path.py
@@ -1,42 +1,140 @@
 import logging
 from pathlib import Path
+import site
+import sys
+import textwrap
+from typing import Tuple, Optional
 
 import userpath  # type: ignore
 from pipx.emojies import stars
+from pipx import constants
 
 
-def ensure_path(location: Path, *, force: bool):
+def wrap_indent(text: str) -> str:
+    text_wrapped = textwrap.wrap(text)
+    if len(text_wrapped) > 1:
+        return (
+            text_wrapped[0]
+            + "\n"
+            + textwrap.indent("\n".join(text_wrapped[1:]), "    ")
+        )
+    else:
+        return text_wrapped[0]
+
+
+def get_pipx_user_bin_path() -> Optional[Path]:
+    """Returns None if pipx is not installed using `pip --user`
+    Otherwise returns parent dir of pipx binary
+    """
+    # NOTE: using this method to detect pip user-installed pipx will return
+    #   None if pipx was installed as editable using `pip install --user -e`
+
+    # https://docs.python.org/3/install/index.html#inst-alt-install-user
+    #   Linux + Mac:
+    #       scripts in <userbase>/bin
+    #   Windows:
+    #       scripts in <userbase>/Python<XY>/Scripts
+    #       modules in <userbase>/Python<XY>/site-packages
+
+    pipx_bin_path = None
+
+    script_path = Path(__file__).resolve()
+    userbase_path = Path(site.getuserbase()).resolve()
+    try:
+        _ = script_path.relative_to(userbase_path)
+    except ValueError:
+        pip_user_installed = False
+    else:
+        pip_user_installed = True
+    if pip_user_installed:
+        test_paths = (
+            userbase_path / "bin" / "pipx",
+            Path(site.getusersitepackages()).resolve().parent / "Scripts" / "pipx.exe",
+        )
+        for test_path in test_paths:
+            if test_path.exists():
+                pipx_bin_path = test_path.parent
+                break
+
+    return pipx_bin_path
+
+
+def ensure_path(location: Path, *, force: bool) -> Tuple[bool, bool]:
+    """Ensure location is in user's PATH or add it to PATH.
+    Returns True if location was added to PATH
+    """
     location_str = str(location)
+    path_added = False
+    need_shell_restart = userpath.need_shell_restart(location_str)
+    in_current_path = userpath.in_current_path(location_str)
+
+    if force or (not in_current_path and not need_shell_restart):
+        userpath.append(location_str)
+        print(
+            wrap_indent(
+                f"Success! Added {location_str} to the PATH environment variable."
+            )
+        )
+        path_added = True
+        need_shell_restart = userpath.need_shell_restart(location_str)
+    elif not in_current_path and need_shell_restart:
+        print(
+            wrap_indent(
+                f"{location_str} has been been added to PATH, but you "
+                "need to open a new terminal or re-login for this PATH "
+                "change to take effect."
+            )
+        )
+    else:
+        print(wrap_indent(f"{location_str} is already in PATH."))
+
+    return (path_added, need_shell_restart)
+
+
+def ensure_pipx_paths(force: bool):
+    bin_paths = [constants.LOCAL_BIN_DIR]
+
+    pipx_user_bin_path = get_pipx_user_bin_path()
+    if pipx_user_bin_path is not None:
+        bin_paths.append(pipx_user_bin_path)
+
+    path_added = False
+    need_shell_restart = False
+    for bin_path in bin_paths:
+        (path_added_current, need_shell_restart_current) = ensure_path(
+            bin_path, force=force
+        )
+        path_added |= path_added_current
+        need_shell_restart |= need_shell_restart_current
 
-    post_install_message = (
-        "You likely need to open a new terminal or re-login for "
-        "the changes to take effect."
-    )
-    if userpath.in_current_path(location_str) or userpath.need_shell_restart(
-        location_str
-    ):
-        if not force:
-            if userpath.need_shell_restart(location_str):
-                print(
-                    f"{location_str} has been already been added to PATH. "
-                    f"{post_install_message}"
-                )
-            else:
-                logging.warning(
-                    (
-                        f"The directory `{location_str}` is already in PATH. If you "
-                        "are sure you want to proceed, try again with "
-                        "the '--force' flag.\n\n"
-                        f"Otherwise pipx is ready to go! {stars}"
-                    )
-                )
-            return
-
-    userpath.append(location_str)
-    print(f"Success! Added {location_str} to the PATH environment variable.")
-    print(
-        "Consider adding shell completions for pipx. "
-        "Run 'pipx completions' for instructions."
-    )
     print()
-    print(f"{post_install_message} {stars}")
+
+    if path_added:
+        print(
+            textwrap.fill(
+                "Consider adding shell completions for pipx. "
+                "Run 'pipx completions' for instructions."
+            )
+            + "\n"
+        )
+    elif not need_shell_restart:
+        sys.stdout.flush()
+        logging.warning(
+            textwrap.fill(
+                "All pipx binary directories have been added to PATH. "
+                "If you are sure you want to proceed, try again with "
+                "the '--force' flag."
+            )
+            + "\n"
+        )
+
+    if need_shell_restart:
+        print(
+            textwrap.fill(
+                "You will need to open a new terminal or re-login for "
+                "the PATH changes to take effect."
+            )
+            + "\n"
+        )
+
+    print(f"Otherwise pipx is ready to go! {stars}")
diff --git a/src/pipx/main.py b/src/pipx/main.py
index 1adfcb6..65bbc2c 100644
--- a/src/pipx/main.py
+++ b/src/pipx/main.py
@@ -205,7 +205,7 @@ def run_pipx_command(args: argparse.Namespace):  # noqa: C901
         return commands.run_pip(package, venv_dir, args.pipargs, args.verbose)
     elif args.command == "ensurepath":
         try:
-            return commands.ensure_path(constants.LOCAL_BIN_DIR, force=args.force)
+            return commands.ensure_pipx_paths(force=args.force)
         except Exception as e:
             raise PipxError(e)
     elif args.command == "completions":
@@ -484,8 +484,10 @@ def _add_ensurepath(subparsers):
     p = subparsers.add_parser(
         "ensurepath",
         help=(
-            "Ensure directory where pipx stores apps is on your "
-            "PATH environment variable. Note that running this may modify "
+            "Ensure directory where pipx stores apps is in your "
+            "PATH environment variable. Also if pipx was installed via "
+            "`pip install --user`, ensure pipx itself is in your PATH. "
+            "Note that running this may modify "
             "your shell's configuration file(s) such as '~/.bashrc'."
         ),
     )
@@ -495,7 +497,7 @@ def _add_ensurepath(subparsers):
         action="store_true",
         help=(
             "Add text to your shell's config file even if it looks like your "
-            f"PATH already has {str(constants.LOCAL_BIN_DIR)}"
+            "PATH already contains paths to pipx and pipx-install apps."
         ),
     )
 
diff --git a/src/pipx/package_specifier.py b/src/pipx/package_specifier.py
new file mode 100644
index 0000000..500a1e4
--- /dev/null
+++ b/src/pipx/package_specifier.py
@@ -0,0 +1,93 @@
+# Valid package specifiers for pipx:
+#   PEP508-compliant
+#   git+<URL>
+#   <URL>
+#   <pypi_package_name>
+#   <pypi_package_name><version_specifier>
+
+import logging
+from pathlib import Path
+from typing import Optional
+
+from packaging.requirements import InvalidRequirement, Requirement
+from packaging.utils import canonicalize_name
+
+from pipx.util import PipxError
+
+
+def parse_specifier_for_metadata(package_spec: str) -> str:
+    """Return package_or_url suitable for pipx metadata
+
+    Specifically:
+    * Strip any version specifiers (e.g. package == 1.5.4)
+    * Strip any markers (e.g. python_version > 3.4)
+    * Convert local paths to absolute paths
+    """
+    # NOTE: If package_spec is valid pypi name, pip will always treat it as a
+    #       pypi package, not checking for local path.
+    #       We replicate pypi precedence here (only non-valid-pypi names
+    #       initiate check for local path, e.g. './package-name')
+
+    valid_pep508 = False
+    valid_url = False
+    valid_local_path = False
+    package_or_url = ""
+
+    try:
+        package_req = Requirement(package_spec)
+    except InvalidRequirement:
+        # not a valid PEP508 package specification
+        valid_pep508 = False
+    else:
+        # valid PEP508 package specification
+        valid_pep508 = True
+        if package_req.url:
+            package_or_url = package_req.url
+        else:
+            if package_req.extras:
+                package_or_url = canonicalize_name(
+                    package_req.name + "[" + ",".join(sorted(package_req.extras)) + "]"
+                )
+            else:
+                package_or_url = canonicalize_name(package_req.name)
+
+    # NOTE: packaging currently (2020-07-19) does basic syntax checks on the URL.
+    #   Some examples of what it will not catch:
+    #       - invalid RCS string (e.g. "gat+https://...")
+    #       - non-existent scheme (e.g. "zzzzz://...")
+    if not valid_pep508:
+        try:
+            package_req = Requirement("notapackagename @ " + package_spec)
+        except InvalidRequirement:
+            valid_url = False
+        else:
+            valid_url = True
+            package_or_url = package_spec
+
+    if not valid_pep508 and not valid_url:
+        package_path = Path(package_spec)
+        try:
+            package_path_exists = package_path.exists()
+        except OSError:
+            package_path_exists = False
+
+        if package_path_exists:
+            valid_local_path = True
+            package_or_url = str(package_path.resolve())
+
+    if not valid_pep508 and not valid_url and not valid_local_path:
+        raise PipxError(f"Unable to parse package spec: {package_spec}")
+
+    logging.info(f"cleaned package spec: {package_or_url}")
+
+    return package_or_url
+
+
+def valid_pypi_name(package_spec: str) -> Optional[str]:
+    try:
+        package_req = Requirement(package_spec)
+    except InvalidRequirement:
+        # not a valid PEP508 package specification
+        return None
+
+    return package_req.name
diff --git a/src/pipx/shared_libs.py b/src/pipx/shared_libs.py
index 9a375c2..43231b5 100644
--- a/src/pipx/shared_libs.py
+++ b/src/pipx/shared_libs.py
@@ -32,7 +32,9 @@ class _SharedLibs:
         if not self.is_valid:
             with animate("creating shared libraries", not verbose):
                 run([DEFAULT_PYTHON, "-m", "venv", "--clear", self.root])
-            self.upgrade(pip_args, verbose)
+            # ignore installed packages to ensure no unexpected patches from the OS vendor
+            # are used
+            self.upgrade(["--ignore-installed"] + pip_args, verbose)
 
     @property
     def is_valid(self):
diff --git a/src/pipx/util.py b/src/pipx/util.py
index d21b0df..03709c4 100644
--- a/src/pipx/util.py
+++ b/src/pipx/util.py
@@ -1,11 +1,10 @@
 import logging
 import os
-import re
 import shutil
 import subprocess
 import sys
 from pathlib import Path
-from typing import List, Sequence, Tuple, Union, Optional
+from typing import List, Optional, Sequence, Tuple, Union
 
 from pipx.constants import WINDOWS
 
@@ -137,13 +136,6 @@ def run(cmd: Sequence[Union[str, Path]], check=True) -> int:
     return returncode
 
 
-def valid_pypi_name(package_name: str) -> bool:
-    # https://www.python.org/dev/peps/pep-0508/#names
-    return bool(
-        re.search(r"^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$", package_name, re.I)
-    )
-
-
 def full_package_description(package: str, package_spec: str) -> str:
     if package == package_spec:
         return package
diff --git a/src/pipx/venv.py b/src/pipx/venv.py
index c9a3b64..d0c67e9 100644
--- a/src/pipx/venv.py
+++ b/src/pipx/venv.py
@@ -1,14 +1,13 @@
 import json
 import logging
 import pkgutil
-import re
-import urllib.parse
 from pathlib import Path
-from typing import Generator, List, NamedTuple, Dict, Set, Optional
+from typing import Dict, Generator, List, NamedTuple, Optional, Set
 
 from pipx.animate import animate
 from pipx.constants import DEFAULT_PYTHON, PIPX_SHARED_PTH
-from pipx.pipx_metadata_file import PipxMetadata, PackageInfo
+from pipx.package_specifier import parse_specifier_for_metadata
+from pipx.pipx_metadata_file import PackageInfo, PipxMetadata
 from pipx.shared_libs import shared_libs
 from pipx.util import (
     PipxError,
@@ -18,10 +17,8 @@ from pipx.util import (
     rmdir,
     run,
     run_subprocess,
-    valid_pypi_name,
 )
 
-
 venv_metadata_inspector_raw = pkgutil.get_data("pipx", "venv_metadata_inspector.py")
 assert venv_metadata_inspector_raw is not None, (
     "pipx could not find required file venv_metadata_inspector.py. "
@@ -288,7 +285,7 @@ class Venv:
         venv_package_metadata = self.get_venv_metadata_for_package(package)
         package_info = PackageInfo(
             package=package,
-            package_or_url=abs_path_if_local(package_or_url, self, pip_args),
+            package_or_url=parse_specifier_for_metadata(package_or_url),
             pip_args=pip_args,
             include_apps=include_apps,
             include_dependencies=include_dependencies,
@@ -365,61 +362,3 @@ class Venv:
         if not self.verbose:
             cmd.append("-q")
         return run(cmd)
-
-
-def abs_path_if_local(package_or_url: str, venv: Venv, pip_args: List[str]) -> str:
-    """Return the absolute path if package_or_url represents a filepath
-    and not a pypi package
-    """
-    # if valid url leave it untouched
-    if urllib.parse.urlparse(package_or_url).scheme:
-        return package_or_url
-
-    pkg_path = Path(package_or_url)
-    if not pkg_path.exists():
-        # no existing path, must be pypi package or non-existent
-        return package_or_url
-
-    # Editable packages are either local or url, non-url must be local.
-    # https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs
-    if "--editable" in pip_args and pkg_path.exists():
-        return str(pkg_path.resolve())
-
-    if not valid_pypi_name(package_or_url):
-        return str(pkg_path.resolve())
-
-    # If all of the above conditions do not return, we may have used a pypi
-    #   package.
-    # If we find a pypi package with this name installed, assume we just
-    #   installed it.
-    pip_search_args: List[str]
-
-    # If user-defined pypi index url, then use it for search
-    try:
-        arg_i = pip_args.index("--index-url")
-    except ValueError:
-        pip_search_args = []
-    else:
-        pip_search_args = pip_args[arg_i : arg_i + 2]
-
-    pip_search_result_str = venv.pip_search(package_or_url, pip_search_args)
-    pip_search_results = pip_search_result_str.split("\n")
-
-    # Get package_or_url and following related lines from pip search stdout
-    pkg_found = False
-    pip_search_found = []
-    for pip_search_line in pip_search_results:
-        if pkg_found:
-            if re.search(r"^\s", pip_search_line):
-                pip_search_found.append(pip_search_line)
-            else:
-                break
-        elif pip_search_line.startswith(package_or_url):
-            pip_search_found.append(pip_search_line)
-            pkg_found = True
-    pip_found_str = " ".join(pip_search_found)
-
-    if pip_found_str.startswith(package_or_url) and "INSTALLED" in pip_found_str:
-        return package_or_url
-    else:
-        return str(pkg_path.resolve())


Test patch:
diff --git a/tests/test_animate.py b/tests/test_animate.py
index 41c3ba3..9176a49 100644
--- a/tests/test_animate.py
+++ b/tests/test_animate.py
@@ -11,16 +11,45 @@ from pipx.animate import (
 
 
 def check_animate_output(
-    capsys, test_string, frame_strings, frame_period, frames_to_test
+    capsys,
+    test_string,
+    frame_strings,
+    frame_period,
+    frames_to_test,
+    extra_animate_time=0.4,
+    extra_after_thread_time=0.1,
 ):
+    # NOTE: extra_animate_time <= 0.3 failed on macos
+    #       extra_after_thread_time <= 0.0 failed on macos
     expected_string = "".join(frame_strings)
 
     chars_to_test = 1 + len("".join(frame_strings[:frames_to_test]))
 
     with pipx.animate.animate(test_string, do_animation=True):
-        time.sleep(frame_period * (frames_to_test - 1) + 0.5)
+        time.sleep(frame_period * (frames_to_test - 1) + extra_animate_time)
+    # Wait before capturing stderr to ensure animate thread is finished
+    #   and to capture all its characters. If some are left over they can cause
+    #   false fail in the next call of check_animate_output()
+    time.sleep(extra_after_thread_time)
     captured = capsys.readouterr()
 
+    print("check_animate_output() Test Debug Output:")
+    if len(captured.err) < chars_to_test:
+        print(
+            "Not enough captured characters--Likely need to increase extra_animate_time"
+        )
+    if (
+        captured.err[: len(frame_strings[0])]
+        != expected_string[: len(frame_strings[0])]
+    ):
+        print("Error in first frame--Might need to increase extra_after_thread_time")
+    print(f"captured characters: {len(captured.err)}")
+    print(f"chars_to_test: {chars_to_test}")
+    for i in range(0, chars_to_test, 40):
+        i_end = min(i + 40, chars_to_test)
+        print(f"expected_string[{i}:{i_end}]: {repr(expected_string[i:i_end])}")
+        print(f"captured.err[{i}:{i_end}]   : {repr(captured.err[i:i_end])}")
+
     assert captured.err[:chars_to_test] == expected_string[:chars_to_test]
 
 
diff --git a/tests/test_package_specifier.py b/tests/test_package_specifier.py
new file mode 100644
index 0000000..60acc9b
--- /dev/null
+++ b/tests/test_package_specifier.py
@@ -0,0 +1,66 @@
+from pathlib import Path
+
+import pytest  # type: ignore
+
+from pipx.package_specifier import parse_specifier_for_metadata
+from pipx.util import PipxError
+
+
+# TODO: Make sure git+ works with tests, correct in test_install as well
+@pytest.mark.parametrize(
+    "package_spec_in,package_or_url_correct,valid_spec",
+    [
+        ("pipx", "pipx", True),
+        ("PiPx_stylized.name", "pipx-stylized-name", True),
+        ("pipx==0.15.0", "pipx", True),
+        ("pipx>=0.15.0", "pipx", True),
+        ("pipx<=0.15.0", "pipx", True),
+        ('pipx;python_version>="3.6"', "pipx", True),
+        ('pipx==0.15.0;python_version>="3.6"', "pipx", True),
+        ("pipx[extra1]", "pipx[extra1]", True),
+        ("pipx[extra1, extra2]", "pipx[extra1,extra2]", True),
+        ("src/pipx", str(Path("src/pipx").resolve()), True),
+        (
+            "git+https://github.com/cs01/nox.git@5ea70723e9e6",
+            "git+https://github.com/cs01/nox.git@5ea70723e9e6",
+            True,
+        ),
+        (
+            "nox@git+https://github.com/cs01/nox.git@5ea70723e9e6",
+            "git+https://github.com/cs01/nox.git@5ea70723e9e6",
+            True,
+        ),
+        (
+            "https://github.com/ambv/black/archive/18.9b0.zip",
+            "https://github.com/ambv/black/archive/18.9b0.zip",
+            True,
+        ),
+        (
+            "black@https://github.com/ambv/black/archive/18.9b0.zip",
+            "https://github.com/ambv/black/archive/18.9b0.zip",
+            True,
+        ),
+        (
+            "black @ https://github.com/ambv/black/archive/18.9b0.zip",
+            "https://github.com/ambv/black/archive/18.9b0.zip",
+            True,
+        ),
+        ("path/doesnt/exist", "non-existent-path", False,),
+        (
+            "https:/github.com/ambv/black/archive/18.9b0.zip",
+            "URL-syntax-error-slash",
+            False,
+        ),
+    ],
+)
+def test_parse_specifier_for_metadata(
+    package_spec_in, package_or_url_correct, valid_spec
+):
+    if valid_spec:
+        package_or_url = parse_specifier_for_metadata(package_spec_in)
+        assert package_or_url == package_or_url_correct
+    else:
+        # print package_spec_in for info in case no error is raised
+        print(f"package_spec_in = {package_spec_in}")
+        with pytest.raises(PipxError, match=r"^Unable to parse package spec"):
+            package_or_url = parse_specifier_for_metadata(package_spec_in)
diff --git a/tests/test_shared_libs.py b/tests/test_shared_libs.py
index 365fecd..5feeddb 100644
--- a/tests/test_shared_libs.py
+++ b/tests/test_shared_libs.py
@@ -10,8 +10,8 @@ now = time.time()
 @pytest.mark.parametrize(
     "mtime,needs_upgrade",
     [
-        (now - shared_libs.SHARED_LIBS_MAX_AGE_SEC - 600, True),
-        (now - shared_libs.SHARED_LIBS_MAX_AGE_SEC + 600, False),
+        (now - shared_libs.SHARED_LIBS_MAX_AGE_SEC - 20 * 60, True),
+        (now - shared_libs.SHARED_LIBS_MAX_AGE_SEC + 20 * 60, False),
     ],
 )
 def test_auto_update_shared_libs(capsys, pipx_temp_env, mtime, needs_upgrade):