Repository: python-poetry/poetry

Issue:
Encoding failure is not caught and propagated
- **Poetry version**:  `1.2.1` and `1.3.0.dev01`
- **Python version**: `3.9.13`
- **OS version and name**: `Windows 11 Enterprise (build 22000.856)`
- **pyproject.toml**: 

```toml
[tool.poetry]
name = "test-poetry-master"
version = "0.1.0"
description = ""
authors = ["kristang"]
readme = "README.md"

[[tool.poetry.source]]
name = "private-pypi"
url = "https://private/pypi/private-pypi/simple/"
secondary = true

[tool.poetry.dependencies]
python = "^3.9"

[tool.poetry.dev-dependencies]
pytest = "^7.1.1"
mypy = "^0.982"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
```


- [x] I am on the [latest](https://github.com/python-poetry/poetry/releases/latest) stable Poetry version, installed using a recommended method.
- [x] I have searched the [issues](https://github.com/python-poetry/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] I have consulted the [FAQ](https://python-poetry.org/docs/faq/) and [blog](https://python-poetry.org/blog/) for any relevant entries or release notes.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option) and have included the output below.
#### Additionally
- [x] Discussed the issue  with @neersighted as well as @finswimmer and @radoering on the Poetry Discord.
## Issue

When adding a secondary, private source and attempting to authenticate via basic-http, `poetry` silently ignores a `UnicodeEncodingError` originating from a failed latin1 encoding of a `str`. 

Instead, poetry moves on to the public [pypi.org](https://pypi.org/) feed and informs the user that the searched package is not available - even if it is available on [pypi.org](https://pypi.org/)

The error stems from loading in a user/password from Windows Certificate Manager which appears to be corrupted and cannot be encoded with 'latin1'

The source of the encoding error is `poetry` invoking `_basic_auth_str` from `auth.py` in the `requests` package:
https://github.com/psf/requests/blob/7104ad4b135daab0ed19d8e41bd469874702342b/requests/auth.py#L56-L60

The originating call in `poetry` is `request` in `utils.authenticator`. 

https://github.com/python-poetry/poetry/blob/0ffe91c0c8e2edde987cf2213d402614a9c2a6a6/src/poetry/utils/authenticator.py#L213-L216

`solve()` has a fairly wide try-catch with a base `Exception`

https://github.com/python-poetry/poetry/blob/0ffe91c0c8e2edde987cf2213d402614a9c2a6a6/src/poetry/mixology/version_solver.py#L110-L123

## Reproducing
As this involves a broken `str` that fails to encode reproducibility is a bit hard. I will try and create tests and create a PR if my skills are sufficient enough to create something useful.

Running `pipenv lock -vvv` yields:

```Python-traceback
  Stack trace:

  4  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:157 in _solve
      155│ 
      156│         try:
    → 157│             result = resolve_version(self._package, self._provider)
      158│ 
      159│             packages = result.packages

  3  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\__init__.py:18 in resolve_version
       16│     solver = VersionSolver(root, provider)
       17│ 
    →  18│     return solver.solve()
       19│ 

  2  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:113 in solve
      111│             next: str | None = self._root.name
      112│             while next is not None:
    → 113│                 self._propagate(next)
      114│                 next = self._choose_package_version()
      115│ 

  1  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:152 in _propagate
      150│                     # where that incompatibility will allow us to derive new assignments
      151│                     # that avoid the conflict.
    → 152│                     root_cause = self._resolve_conflict(incompatibility)
      153│ 
      154│                     # Back jumping erases all the assignments we did at the previous

  SolveFailure

  Because test-poetry-master depends on mypy (^0.982) which doesn't exist, version solving failed.

  at ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:351 in _resolve_conflict
      347│             )
      348│             self._log(f'! which is caused by "{most_recent_satisfier.cause}"')
      349│             self._log(f"! thus: {incompatibility}")
      350│ 
    → 351│         raise SolveFailure(incompatibility)
      352│ 
      353│     def _choose_package_version(self) -> str | None:
      354│         """
      355│         Tries to select a version of a required package.

The following error occurred when trying to handle this error:


  Stack trace:

  11  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:329 in run
       327│ 
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  10  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\application.py:185 in _run
       183│         self._load_plugins(io)
       184│ 
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│ 

   9  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│ 
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│ 

   8  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:465 in _run_command
       463│ 
       464│         if error is not None:
     → 465│             raise error
       466│ 
       467│         return event.exit_code

   7  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:449 in _run_command
       447│ 
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   6  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

   5  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\command.py:83 in execute
        81│ 
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

   4  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\commands\lock.py:54 in handle
        52│         self.installer.lock(update=not self.option("no-update"))
        53│ 
     →  54│         return self.installer.run()
        55│ 

   3  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:114 in run
       112│             self._execute_operations = False
       113│ 
     → 114│         return self._do_install()
       115│ 
       116│     def dry_run(self, dry_run: bool = True) -> Installer:

   2  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:247 in _do_install
       245│                 source_root=self._env.path.joinpath("src")
       246│             ):
     → 247│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       248│         else:
       249│             self._io.write_line("Installing dependencies from lock file")

   1  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:74 in solve
        72│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        73│             start = time.time()
     →  74│             packages, depths = self._solve()
        75│             end = time.time()
        76│ 

  SolverProblemError

  Because test-poetry-master depends on mypy (^0.982) which doesn't exist, version solving failed.

  at ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:163 in _solve
      159│             packages = result.packages
      160│         except OverrideNeeded as e:
      161│             return self._solve_in_compatibility_mode(e.overrides)
      162│         except SolveFailure as e:
    → 163│             raise SolverProblemError(e)
      164│ 
      165│         combined_nodes = depth_first_search(PackageNode(self._package, packages))
      166│         results = dict(aggregate_package_nodes(nodes) for nodes in combined_nodes)
      167│
```
No-where in the stacktrace is a user informed that the underlying issue is an encoding failure of credentials.

### Pseduo fix

Wrapping `requests.auth.HTTPBasicAuth` in a try-except can catch the `UnicodeEncodeError` and propagate it to the user: 

```py
if credential.username is not None or credential.password is not None:
    try:
        request = requests.auth.HTTPBasicAuth(
            credential.username or "", credential.password or ""
        )(request)
    except UnicodeEncodeError as e:
        raise PoetryException(e)
```
The new stacktrace now contains the correct `UnicodeEncodeError`:

```Python-traceback
  Stack trace:

  2  ~\Documents\Projects\GITHUB\poetry\src\poetry\utils\authenticator.py:216 in request
      214│         if credential.username is not None or credential.password is not None:
      215│             try:
    → 216│                 request = requests.auth.HTTPBasicAuth(
      217│                     credential.username or "", credential.password or ""
      218│                 )(request)

  1  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\requests\auth.py:95 in __call__
       93│ 
       94│     def __call__(self, r):
    →  95│         r.headers["Authorization"] = _basic_auth_str(self.username, self.password)
       96│         return r
       97│ 

  UnicodeEncodeError

  'latin-1' codec can't encode characters in position 0-5: ordinal not in range(256)

  at ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\requests\auth.py:60 in _basic_auth_str
       56│     if isinstance(username, str):
       57│         username = username.encode("latin1")
       58│ 
       59│     if isinstance(password, str):
    →  60│         password = password.encode("latin1")
       61│ 
       62│     authstr = "Basic " + to_native_string(
       63│         b64encode(b":".join((username, password))).strip()
       64│     )

The following error occurred when trying to handle this error:


  Stack trace:

  24  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:329 in run
       327│ 
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  23  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\application.py:185 in _run
       183│         self._load_plugins(io)
       184│ 
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│ 

  22  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│ 
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│ 

  21  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:465 in _run_command
       463│ 
       464│         if error is not None:
     → 465│             raise error
       466│ 
       467│         return event.exit_code

  20  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:449 in _run_command
       447│ 
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  19  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

  18  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\command.py:83 in execute
        81│ 
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

  17  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\commands\lock.py:54 in handle
        52│         self.installer.lock(update=not self.option("no-update"))
        53│ 
     →  54│         return self.installer.run()
        55│ 

  16  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:114 in run
       112│             self._execute_operations = False
       113│ 
     → 114│         return self._do_install()
       115│ 
       116│     def dry_run(self, dry_run: bool = True) -> Installer:

  15  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:247 in _do_install
       245│                 source_root=self._env.path.joinpath("src")
       246│             ):
     → 247│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       248│         else:
       249│             self._io.write_line("Installing dependencies from lock file")

  14  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:74 in solve
        72│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        73│             start = time.time()
     →  74│             packages, depths = self._solve()
        75│             end = time.time()
        76│ 

  13  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:157 in _solve
       155│ 
       156│         try:
     → 157│             result = resolve_version(self._package, self._provider)
       158│ 
       159│             packages = result.packages

  12  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\__init__.py:18 in resolve_version
        16│     solver = VersionSolver(root, provider)
        17│ 
     →  18│     return solver.solve()
        19│ 

  11  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:114 in solve
       112│             while next is not None:
       113│                 self._propagate(next)
     → 114│                 next = self._choose_package_version()
       115│ 
       116│             return self._result()

  10  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:413 in _choose_package_version
       411│             dependency = unsatisfied[0]
       412│         else:
     → 413│             dependency = min(*unsatisfied, key=_get_min)
       414│ 
       415│         locked = self._provider.get_locked(dependency)

   9  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:399 in _get_min
       397│ 
       398│             try:
     → 399│                 num_packages = len(self._dependency_cache.search_for(dependency))
       400│             except ValueError:
       401│                 num_packages = 0

   8  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:62 in _search_for
        60│         packages = self.cache.get(key)
        61│         if packages is None:
     →  62│             packages = self.provider.search_for(dependency)
        63│         else:
        64│             packages = [

   7  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\provider.py:316 in search_for
        314│             return PackageCollection(dependency, packages)
        315│ 
     →  316│         packages = self._pool.find_packages(dependency)
        317│ 
        318│         packages.sort(

   6  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\pool.py:136 in find_packages
       134│         packages: list[Package] = []
       135│         for repo in self.repositories:
     → 136│             packages += repo.find_packages(dependency)
       137│         return packages
       138│ 

   5  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\repository.py:42 in find_packages
        40│         ignored_pre_release_packages = []
        41│ 
     →  42│         for package in self._find_packages(dependency.name, constraint):
        43│             if package.yanked and not isinstance(constraint, Version):
        44│                 # PEP 592: yanked files are always ignored, unless they are the only

   4  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\legacy_repository.py:96 in _find_packages
        94│             versions = self._cache.store("matches").get(key)
        95│         else:
     →  96│             page = self._get_page(f"/{name}/")
        97│             if page is None:
        98│                 self._log(

   3  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\legacy_repository.py:150 in _get_page
       148│     def _get_page(self, endpoint: str) -> SimpleRepositoryPage | None:
       149│         try:
     → 150│             response = self._get_response(endpoint)
       151│         except UnicodeEncodeError as e:
       152│             raise e

   2  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\http.py:272 in _get_response
       270│         url = self._url + endpoint
       271│         try:
     → 272│             response: requests.Response = self.session.get(
       273│                 url, raise_for_status=False, timeout=REQUESTS_TIMEOUT
       274│             )

   1  ~\Documents\Projects\GITHUB\poetry\src\poetry\utils\authenticator.py:274 in get
       272│ 
       273│     def get(self, url: str, **kwargs: Any) -> requests.Response:
     → 274│         return self.request("get", url, **kwargs)
       275│ 
       276│     def post(self, url: str, **kwargs: Any) -> requests.Response:

  PoetryException

  'latin-1' codec can't encode characters in position 0-5: ordinal not in range(256)

  at ~\Documents\Projects\GITHUB\poetry\src\poetry\utils\authenticator.py:220 in request
      216│                 request = requests.auth.HTTPBasicAuth(
      217│                     credential.username or "", credential.password or ""
      218│                 )(request)
      219│             except UnicodeEncodeError as e:
    → 220│                 raise PoetryException(e)
      221│ 
      222│         session = self.get_session(url=url)
      223│         prepared_request = session.prepare_request(request)
      224│ 


```




Discussion:
> solve() has a fairly wide try-catch with a base Exception

but since it re-raises that exception that's not masking anything, surely?  Seems plausible that there's a too-broad-except somewhere, but not that one?
@dimbleby: Yes, it might be somewhere else. That was just some extra info.
extra, but I think not relevant!

https://github.com/python-poetry/poetry/blob/0ffe91c0c8e2edde987cf2213d402614a9c2a6a6/src/poetry/mixology/version_solver.py#L400 looks like a more plausible candidate to me, or possibly one of the other places where poetry catches `ValueError` - that would capture a unicode error.
so far as I can see that try-except can simply be removed, which should allow your error to propagate.  If you fancy submitting an MR, I reckon that's the way to go
I'm not that experienced in best-practice in terms of changes like this, but it seems a bit dangerous to just remove a try-except like that - but if you feel confident in doing so I have created PR #6786.

 

Resolving patch:
diff --git a/src/poetry/mixology/incompatibility.py b/src/poetry/mixology/incompatibility.py
--- a/src/poetry/mixology/incompatibility.py
+++ b/src/poetry/mixology/incompatibility.py
@@ -5,7 +5,6 @@
 from poetry.mixology.incompatibility_cause import ConflictCause
 from poetry.mixology.incompatibility_cause import DependencyCause
 from poetry.mixology.incompatibility_cause import NoVersionsCause
-from poetry.mixology.incompatibility_cause import PackageNotFoundCause
 from poetry.mixology.incompatibility_cause import PlatformCause
 from poetry.mixology.incompatibility_cause import PythonCause
 from poetry.mixology.incompatibility_cause import RootCause
@@ -146,11 +145,6 @@ def __str__(self) -> str:
                 f"no versions of {self._terms[0].dependency.name} match"
                 f" {self._terms[0].constraint}"
             )
-        elif isinstance(self._cause, PackageNotFoundCause):
-            assert len(self._terms) == 1
-            assert self._terms[0].is_positive()
-
-            return f"{self._terms[0].dependency.name} doesn't exist"
         elif isinstance(self._cause, RootCause):
             assert len(self._terms) == 1
             assert not self._terms[0].is_positive()
@@ -420,8 +414,6 @@ def _try_requires_forbidden(
             buffer.append(f"which requires Python {cause.python_version}")
         elif isinstance(latter.cause, NoVersionsCause):
             buffer.append("which doesn't match any versions")
-        elif isinstance(latter.cause, PackageNotFoundCause):
-            buffer.append("which doesn't exist")
         else:
             buffer.append("which is forbidden")
 
diff --git a/src/poetry/mixology/incompatibility_cause.py b/src/poetry/mixology/incompatibility_cause.py
--- a/src/poetry/mixology/incompatibility_cause.py
+++ b/src/poetry/mixology/incompatibility_cause.py
@@ -79,17 +79,3 @@ def __init__(self, platform: str) -> None:
     @property
     def platform(self) -> str:
         return self._platform
-
-
-class PackageNotFoundCause(IncompatibilityCause):
-    """
-    The incompatibility represents a package that couldn't be found by its
-    source.
-    """
-
-    def __init__(self, error: Exception) -> None:
-        self._error = error
-
-    @property
-    def error(self) -> Exception:
-        return self._error
diff --git a/src/poetry/mixology/version_solver.py b/src/poetry/mixology/version_solver.py
--- a/src/poetry/mixology/version_solver.py
+++ b/src/poetry/mixology/version_solver.py
@@ -3,7 +3,6 @@
 import functools
 import time
 
-from contextlib import suppress
 from typing import TYPE_CHECKING
 
 from poetry.core.packages.dependency import Dependency
@@ -12,7 +11,6 @@
 from poetry.mixology.incompatibility import Incompatibility
 from poetry.mixology.incompatibility_cause import ConflictCause
 from poetry.mixology.incompatibility_cause import NoVersionsCause
-from poetry.mixology.incompatibility_cause import PackageNotFoundCause
 from poetry.mixology.incompatibility_cause import RootCause
 from poetry.mixology.partial_solution import PartialSolution
 from poetry.mixology.result import SolverResult
@@ -395,10 +393,8 @@ def _get_min(dependency: Dependency) -> tuple[bool, int, int]:
                 if locked:
                     return is_specific_marker, Preference.LOCKED, 1
 
-            try:
-                num_packages = len(self._dependency_cache.search_for(dependency))
-            except ValueError:
-                num_packages = 0
+            num_packages = len(self._dependency_cache.search_for(dependency))
+
             if num_packages < 2:
                 preference = Preference.NO_CHOICE
             elif use_latest:
@@ -414,28 +410,8 @@ def _get_min(dependency: Dependency) -> tuple[bool, int, int]:
 
         locked = self._provider.get_locked(dependency)
         if locked is None:
-            try:
-                packages = self._dependency_cache.search_for(dependency)
-            except ValueError as e:
-                self._add_incompatibility(
-                    Incompatibility([Term(dependency, True)], PackageNotFoundCause(e))
-                )
-                complete_name: str = dependency.complete_name
-                return complete_name
-
-            package = None
-            if locked is not None:
-                package = next(
-                    (
-                        p
-                        for p in packages
-                        if p.package.version == locked.package.version
-                    ),
-                    None,
-                )
-            if package is None:
-                with suppress(IndexError):
-                    package = packages[0]
+            packages = self._dependency_cache.search_for(dependency)
+            package = next(iter(packages), None)
 
             if package is None:
                 # If there are no versions that satisfy the constraint,


Test patch: