Repository: trailofbits/pip-audit (python)

Issue:
does not work when one of many dependencíes is not present on PyPI
## Bug description

Running this tool on a requirements.txt where one of the many entries e.g. points to github causes an error.

## Reproduction steps 

A step-by-step list of actions to reproduce the behavior.

```
git clone https://git.launchpad.net/lpcraft

cd lpcraft

pip-audit -r requirements.txt

/ Resolving craft-cli                             
Traceback (most recent call last):
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/resolvelib.py", line 41, in resolve
    result = self.resolver.resolve([req])
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/resolvelib/resolvers.py", line 348, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/resolvelib/resolvers.py", line 147, in _add_to_criteria
    matches = self._p.find_matches(
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 279, in find_matches
    return sorted(candidates, key=attrgetter("version", "is_wheel"), reverse=True)
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 272, in <genexpr>
    candidates = (
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 188, in get_project_from_pypi
    response.raise_for_status()
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/simple/craft-cli/

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/requirement.py", line 62, in collect
    for _, deps in self.resolver.resolve_all(iter(req_values)):
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/interface.py", line 67, in resolve_all
    yield (req, self.resolve(req))
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/resolvelib.py", line 43, in resolve
    raise ResolveLibResolverError("failed to resolve dependencies") from e
pip_audit._dependency_source.resolvelib.resolvelib.ResolveLibResolverError: failed to resolve dependencies

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jugmac00/.local/bin/pip-audit", line 8, in <module>
    sys.exit(audit())
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_cli.py", line 240, in audit
    for (spec, vulns) in auditor.audit(source):
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_audit.py", line 60, in audit
    yield from self._service.query_all(specs)
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_service/interface.py", line 115, in query_all
    for spec in specs:
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/requirement.py", line 74, in collect
    raise RequirementSourceError("dependency resolver raised an error") from dre
pip_audit._dependency_source.requirement.RequirementSourceError: dependency resolver raised an error
```

## Expected behavior

I expected to receive a proper report.

## Screenshots and logs

```
$ PIP_AUDIT_LOGLEVEL=debug pip-audit -r requirements.txt 
DEBUG:pip_audit._cli:parsed arguments: Namespace(cache_dir=None, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, dry_run=False, format=<OutputFormatChoice.Columns: 'columns'>, local=False, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, requirements=[<_io.TextIOWrapper name='requirements.txt' mode='r' encoding='UTF-8'>], strict=False, timeout=15, vulnerability_service=<VulnerabilityServiceChoice.Pypi: 'pypi'>)
/ Resolving craft-cli                             
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/certifi HTTP/1.1" 301 113
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/certifi/ HTTP/1.1" 200 7534
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): files.pythonhosted.org:443
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/37/45/946c02767aabb873146011e665728b680884cd8fe70dde973c640e45b775/certifi-2021.10.8-py2.py3-none-any.whl HTTP/1.1" 200 149195
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/pypi/certifi/2021.10.8/json" in the cache
DEBUG:cachecontrol.controller:Current age based on date: 324
DEBUG:cachecontrol.controller:Freshness lifetime from max-age: 900
DEBUG:cachecontrol.controller:The response is "fresh", returning cached response
DEBUG:cachecontrol.controller:900 > 324
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/charset-normalizer HTTP/1.1" 301 224
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/charset-normalizer/ HTTP/1.1" 200 5124
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): files.pythonhosted.org:443
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/de/c8/820b1546c68efcbbe3c1b10dd925fbd84a0dda7438bc18db0ef1fa567733/charset_normalizer-2.0.7-py3-none-any.whl HTTP/1.1" 200 38247
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/pypi/charset-normalizer/2.0.7/json" in the cache
DEBUG:cachecontrol.controller:Current age based on date: 324
DEBUG:cachecontrol.controller:Freshness lifetime from max-age: 900
DEBUG:cachecontrol.controller:The response is "fresh", returning cached response
DEBUG:cachecontrol.controller:900 > 324
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/craft-cli HTTP/1.1" 301 215
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/craft-cli/ HTTP/1.1" 404 13
Traceback (most recent call last):
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/resolvelib.py", line 41, in resolve
    result = self.resolver.resolve([req])
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/resolvelib/resolvers.py", line 348, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/resolvelib/resolvers.py", line 147, in _add_to_criteria
    matches = self._p.find_matches(
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 279, in find_matches
    return sorted(candidates, key=attrgetter("version", "is_wheel"), reverse=True)
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 272, in <genexpr>
    candidates = (
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 188, in get_project_from_pypi
    response.raise_for_status()
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/simple/craft-cli/

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/requirement.py", line 62, in collect
    for _, deps in self.resolver.resolve_all(iter(req_values)):
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/interface.py", line 67, in resolve_all
    yield (req, self.resolve(req))
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/resolvelib.py", line 43, in resolve
    raise ResolveLibResolverError("failed to resolve dependencies") from e
pip_audit._dependency_source.resolvelib.resolvelib.ResolveLibResolverError: failed to resolve dependencies

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jugmac00/.local/bin/pip-audit", line 8, in <module>
    sys.exit(audit())
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_cli.py", line 240, in audit
    for (spec, vulns) in auditor.audit(source):
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_audit.py", line 60, in audit
    yield from self._service.query_all(specs)
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_service/interface.py", line 115, in query_all
    for spec in specs:
  File "/home/jugmac00/.local/pipx/venvs/pip-audit/lib/python3.8/site-packages/pip_audit/_dependency_source/requirement.py", line 74, in collect
    raise RequirementSourceError("dependency resolver raised an error") from dre
pip_audit._dependency_source.requirement.RequirementSourceError: dependency resolver raised an error

```

## Platform information

* OS name and version: Ubuntu 20.4
* `pip-audit` version (`pip-audit -V`):  1.0.0
* Python version (`python -V` or `python3 -V`):  3.8.10
* `pip` version (`pip -V` or `pip3 -V`): N/A


Pull request description:
_dependency_source: Return a `SkippedDependency` when the resolver can't find a project on PyPI
Closes #157

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7fd18b5..f9f0aa5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,6 +18,12 @@ All versions prior to 0.0.9 are untracked.
   as a bare option, `--desc` is equivalent to `--desc on`
   ([#153](https://github.com/trailofbits/pip-audit/pull/153))
 
+* Dependency resolution: The PyPI-based dependency resolver no longer throws
+  an uncaught exception on package resolution errors; instead, the package
+  is marked as skipped and an appropriate warning or fatal error (in
+  `--strict` mode) is produced
+  ([#162](https://github.com/trailofbits/pip-audit/pull/162))
+
 ### Removed
 
 ## [1.0.0] - 2021-12-1
diff --git a/pip_audit/_dependency_source/interface.py b/pip_audit/_dependency_source/interface.py
index 1930567..13f49c3 100644
--- a/pip_audit/_dependency_source/interface.py
+++ b/pip_audit/_dependency_source/interface.py
@@ -8,7 +8,7 @@ from typing import Iterator, List, Tuple
 
 from packaging.requirements import Requirement
 
-from pip_audit._service import Dependency, ResolvedDependency
+from pip_audit._service import Dependency
 
 
 class DependencySource(ABC):
@@ -48,15 +48,15 @@ class DependencyResolver(ABC):
     """
 
     @abstractmethod
-    def resolve(self, req: Requirement) -> List[ResolvedDependency]:  # pragma: no cover
+    def resolve(self, req: Requirement) -> List[Dependency]:  # pragma: no cover
         """
-        Resolve a single `Requirement` into a list of concrete `Dependency` instances.
+        Resolve a single `Requirement` into a list of `Dependency` instances.
         """
         raise NotImplementedError
 
     def resolve_all(
         self, reqs: Iterator[Requirement]
-    ) -> Iterator[Tuple[Requirement, List[ResolvedDependency]]]:
+    ) -> Iterator[Tuple[Requirement, List[Dependency]]]:
         """
         Resolve a collection of `Requirement`s into their respective `Dependency` sets.
 
diff --git a/pip_audit/_dependency_source/requirement.py b/pip_audit/_dependency_source/requirement.py
index 2db5b48..a7238d6 100644
--- a/pip_audit/_dependency_source/requirement.py
+++ b/pip_audit/_dependency_source/requirement.py
@@ -3,7 +3,7 @@ Collect dependencies from one or more `requirements.txt`-formatted files.
 """
 
 from pathlib import Path
-from typing import Iterator, List, Optional, Set
+from typing import Iterator, List, Optional, Set, cast
 
 from packaging.requirements import Requirement
 from pip_api import parse_requirements
@@ -16,6 +16,7 @@ from pip_audit._dependency_source import (
     DependencySourceError,
 )
 from pip_audit._service import Dependency
+from pip_audit._service.interface import ResolvedDependency, SkippedDependency
 from pip_audit._state import AuditState
 
 
@@ -64,10 +65,13 @@ class RequirementSource(DependencySource):
                         # Don't allow duplicate dependencies to be returned
                         if dep in collected:
                             continue
-                        if self.state is not None:
-                            self.state.update_state(
-                                f"Collecting {dep.name} ({dep.version})"
-                            )  # pragma: no cover
+                        if self.state is not None:  # pragma: no cover
+                            if dep.is_skipped():
+                                dep = cast(SkippedDependency, dep)
+                                self.state.update_state(f"Skipping {dep.name}: {dep.skip_reason}")
+                            else:
+                                dep = cast(ResolvedDependency, dep)
+                                self.state.update_state(f"Collecting {dep.name} ({dep.version})")
                         collected.add(dep)
                         yield dep
             except DependencyResolverError as dre:
diff --git a/pip_audit/_dependency_source/resolvelib/pypi_provider.py b/pip_audit/_dependency_source/resolvelib/pypi_provider.py
index 6532d57..b4e73a6 100644
--- a/pip_audit/_dependency_source/resolvelib/pypi_provider.py
+++ b/pip_audit/_dependency_source/resolvelib/pypi_provider.py
@@ -185,6 +185,8 @@ def get_project_from_pypi(project, extras, timeout: Optional[int], state: Option
     """Return candidates created from the project name and extras."""
     url = "https://pypi.org/simple/{}".format(project)
     response: requests.Response = requests.get(url, timeout=timeout)
+    if response.status_code == 404:
+        raise PyPINotFoundError(f'Could not find project "{project}" on PyPI')
     response.raise_for_status()
     data = response.content
     doc = html5lib.parse(data, namespaceHTMLElements=False)
@@ -291,3 +293,11 @@ class PyPIProvider(AbstractProvider):
         See `resolvelib.providers.AbstractProvider.get_dependencies`.
         """
         return candidate.dependencies
+
+
+class PyPINotFoundError(Exception):
+    """
+    An error to signify that the provider could not find the requested project on PyPI.
+    """
+
+    pass
diff --git a/pip_audit/_dependency_source/resolvelib/resolvelib.py b/pip_audit/_dependency_source/resolvelib/resolvelib.py
index 6392526..665d153 100644
--- a/pip_audit/_dependency_source/resolvelib/resolvelib.py
+++ b/pip_audit/_dependency_source/resolvelib/resolvelib.py
@@ -3,6 +3,7 @@ Resolve a list of dependencies via the `resolvelib` API as well as a custom
 `Resolver` that uses PyPI as an information source.
 """
 
+import logging
 from typing import List, Optional
 
 from packaging.requirements import Requirement
@@ -10,10 +11,12 @@ from requests.exceptions import HTTPError
 from resolvelib import BaseReporter, Resolver
 
 from pip_audit._dependency_source import DependencyResolver, DependencyResolverError
-from pip_audit._service.interface import ResolvedDependency
+from pip_audit._service.interface import Dependency, ResolvedDependency, SkippedDependency
 from pip_audit._state import AuditState
 
-from .pypi_provider import PyPIProvider
+from .pypi_provider import PyPINotFoundError, PyPIProvider
+
+logger = logging.getLogger(__name__)
 
 
 class ResolveLibResolver(DependencyResolver):
@@ -32,13 +35,17 @@ class ResolveLibResolver(DependencyResolver):
         self.reporter = BaseReporter()
         self.resolver: Resolver = Resolver(self.provider, self.reporter)
 
-    def resolve(self, req: Requirement) -> List[ResolvedDependency]:
+    def resolve(self, req: Requirement) -> List[Dependency]:
         """
         Resolve the given `Requirement` into a `Dependency` list.
         """
-        deps: List[ResolvedDependency] = []
+        deps: List[Dependency] = []
         try:
             result = self.resolver.resolve([req])
+        except PyPINotFoundError as e:
+            skip_reason = str(e)
+            logger.debug(skip_reason)
+            return [SkippedDependency(name=req.name, skip_reason=skip_reason)]
         except HTTPError as e:
             raise ResolveLibResolverError("failed to resolve dependencies") from e
         for name, candidate in result.mapping.items():


Test patch:
diff --git a/test/dependency_source/test_resolvelib.py b/test/dependency_source/test_resolvelib.py
index 47188a9..7d81c1c 100644
--- a/test/dependency_source/test_resolvelib.py
+++ b/test/dependency_source/test_resolvelib.py
@@ -10,13 +10,14 @@ from resolvelib.resolvers import InconsistentCandidate, ResolutionImpossible
 
 from pip_audit._dependency_source import resolvelib
 from pip_audit._dependency_source.resolvelib import pypi_provider
-from pip_audit._service.interface import Dependency, ResolvedDependency
+from pip_audit._service.interface import Dependency, ResolvedDependency, SkippedDependency
 
 
 def get_package_mock(data):
     class Doc:
         def __init__(self, content):
             self.content = content
+            self.status_code = 200
 
         def raise_for_status(self):
             pass
@@ -239,6 +240,9 @@ def test_resolvelib_sdist_invalid_suffix(monkeypatch):
 def test_resolvelib_http_error(monkeypatch):
     def get_http_error_mock():
         class Doc:
+            def __init__(self):
+                self.status_code = 400
+
             def raise_for_status(self):
                 raise HTTPError
 
@@ -250,3 +254,24 @@ def test_resolvelib_http_error(monkeypatch):
     req = Requirement("flask==2.0.1")
     with pytest.raises(resolvelib.ResolveLibResolverError):
         dict(resolver.resolve_all([req]))
+
+
+def test_resolvelib_http_notfound(monkeypatch):
+    def get_http_not_found_mock():
+        class Doc:
+            def __init__(self):
+                self.status_code = 404
+
+        return Doc()
+
+    monkeypatch.setattr(requests, "get", lambda _url, **kwargs: get_http_not_found_mock())
+
+    resolver = resolvelib.ResolveLibResolver()
+    req = Requirement("flask==2.0.1")
+    resolved_deps = dict(resolver.resolve_all([req]))
+    assert len(resolved_deps) == 1
+    expected_deps = [
+        SkippedDependency(name="flask", skip_reason='Could not find project "flask" on PyPI')
+    ]
+    assert req in resolved_deps
+    assert resolved_deps[req] == expected_deps