Repository: python-poetry/poetry

Issue:
Poetry hangs when installing libraries depending on `pyparsing` on M1 Apple Silicon
<!--
  Hi there! Thank you for discovering and submitting an issue.

  Before you submit this; let's make sure of a few things.
  Please make sure the following boxes are ticked if they are correct.
  If not, please try and fulfill these first.
-->

<!-- Checked checkbox should look like this: [x] -->
- [x] I am on the [latest](https://github.com/python-poetry/poetry/releases/latest) Poetry version.
- [x] I have searched the [issues](https://github.com/python-poetry/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option).

<!--
  Once those are done, if you're able to fill in the following list with your information,
  it'd be very helpful to whoever handles the issue.
-->

- **OS version and name**: macOS Monterrey 12.4, MacBook Pro (13-inch, M1, 2020)
- **Poetry version**: 1.1.14
- **Link of a [Gist](https://gist.github.com/) with the contents of your pyproject.toml file**: [gist](https://gist.github.com/fernandobperezm/879c9eaab4161693b8d842f185dca71d)

## Issue
<!-- Now feel free to write your issue, but please be descriptive! Thanks again 🙌 ❤️ -->

Hello! I recently changed my intel mac with an M1 one. 
In the process, I noticed that all my poetry installations hang (more than 900secs without finishing) if any library I'm installing depends on `pyparsing (>=2.0.2,!=3.0.5)`.

The followings are the steps to reproduce my environment.
```bash
poetry new pytest-demo
cd pytest-demo 
pyenv local 3.9.7
poetry env use 3.9.7
```

Then I comment the pytest dependency in the default `pyproject.toml` file, it should look like the following:
```toml
[tool.poetry]
name = "pytest-demo"
version = "0.1.0"
description = ""
authors = ["Fernando Benjamin PEREZ MAURERA <fperezmaurera@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.9"

# [tool.poetry.dev-dependencies]
# pytest = "^5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
```


Then I run the following command (using the `-vvv`) option:
```bash
poetry add pytest -vvv -D
```

The output of this command is the following
```bash
Using virtualenv: /Users/fernandobperezm/Library/Caches/pypoetry/virtualenvs/pytest-demo-nBp3JOZu-py3.9
PyPI: 152 packages found for pytest *
Using version ^7.1.2 for pytest

Updating dependencies
Resolving dependencies...
   1: fact: pytest-demo is 0.1.0
   1: derived: pytest-demo
   1: fact: pytest-demo depends on pytest (^7.1.2)
   1: fact: pytest-demo depends on pytest (^7.1.2)
   1: selecting pytest-demo (0.1.0)
   1: derived: pytest (>=7.1.2,<8.0.0)
PyPI: 1 packages found for pytest >=7.1.2,<8.0.0
   1: fact: pytest (7.1.2) depends on attrs (>=19.2.0)
   1: fact: pytest (7.1.2) depends on iniconfig (*)
   1: fact: pytest (7.1.2) depends on packaging (*)
   1: fact: pytest (7.1.2) depends on pluggy (>=0.12,<2.0)
   1: fact: pytest (7.1.2) depends on py (>=1.8.2)
   1: fact: pytest (7.1.2) depends on tomli (>=1.0.0)
   1: fact: pytest (7.1.2) depends on atomicwrites (>=1.0)
   1: fact: pytest (7.1.2) depends on colorama (*)
   1: selecting pytest (7.1.2)
   1: derived: colorama
   1: derived: atomicwrites (>=1.0)
   1: derived: tomli (>=1.0.0)
   1: derived: py (>=1.8.2)
   1: derived: pluggy (>=0.12,<2.0)
   1: derived: packaging
   1: derived: iniconfig
   1: derived: attrs (>=19.2.0)
PyPI: 43 packages found for colorama *
PyPI: 8 packages found for atomicwrites >=1.0
PyPI: 12 packages found for tomli >=1.0.0
PyPI: No release information found for py-0.8.0-alpha2, skipping
PyPI: No release information found for py-0.9.0, skipping
PyPI: No release information found for py-1.4.32.dev1, skipping
PyPI: 4 packages found for py >=1.8.2
PyPI: 4 packages found for pluggy >=0.12,<2.0
PyPI: 39 packages found for packaging *
PyPI: No release information found for iniconfig-0.0, skipping
PyPI: 5 packages found for iniconfig *
PyPI: 9 packages found for attrs >=19.2.0
   1: selecting py (1.11.0)
   1: selecting pluggy (1.0.0)
   1: selecting iniconfig (1.1.1)
   1: selecting attrs (21.4.0)
   1: selecting tomli (2.0.1)
   1: fact: packaging (21.3) depends on pyparsing (>=2.0.2,<3.0.5 || >3.0.5)
   1: selecting packaging (21.3)
   1: derived: pyparsing (>=2.0.2,!=3.0.5)
```

The installation then hangs and never ends (at least not after 900secs which is the max amount of time I've waited). I don't know if this is something related to my python version (3.9.7). This happened as well with other libraries, such as `dask` using `poetry add "dask[complete]"`

If you need more info by running other commands I'm more than happy to help. 



Discussion:
I've previously seen people report that this goes away after they clear their cache.  

However at that point they've cleared their cache and have no means to reproduce the problem and no-one can debug it.

So if you're willing to do a bit of work, it would be lovely if you can

- _move_ your cache somewhere, and see whether that allows things to unstick
- if it does, then gradually reintroduce parts of your moved cache until you can reproduce the issue with as minimal a cache as possible
- then find some way of sharing that - or debug it yourself! - so that someone can investigate

Of course this might not be anything to do with the cache at all, who knows.
Thanks, David!

I tried deleting and re-creating the cache, and that worked.

I also tried to dissect the problem and found a possible culprit: an `_http` folder inside poetry's PyPi cache (dir is: `<poetry cache dir>/cache/repositories/pypi/_http`, in my machine it is located at: `/Users/fernandobperezm/Library/Caches/pypoetry/cache/repositories/pypi/_http`). 

I'm sharing a Google Drive link ([here](https://drive.google.com/drive/folders/1YZajMMGQoKFq12ZsZov9zsERQZp-vFxn?usp=sharing)) with two _http folders. The one suffixed with `old` is the one that causes the issue. The one suffixed with `new` is one created by poetry and led to successfully installing `pytest`.  

Sadly, I do not know about the structure of this folder or its contents, so I'm not able to debug it myself. Hopefully, somebody may have a look at it. If not, I can take a look at it, but I'll need some guidance on how to do so 😅



pretty sure the problem is going to be this lockfile: `/a/4/5/1/5/a4515689376b70d90d7a9b3311690220aedc069190a8454969903b1c.lock`.

I imagine you hit Ctrl-C or similar at some inopportune moment, and that left your cache locked forever.

Ultimately that file will I think come from `filelock` via `cachecontrol`.  

`filelock` is deprecated, but efforts to replace it in `cachecontrol` are not progressing: https://github.com/ionrock/cachecontrol/issues/109.  

I don't know whether it's even possible to write `filelock` or equivalent in a way that doesn't sometimes leak an unwanted lockfile - but in view of the lack of recent movement in `cachecontrol`, we shouldn't expect it to happen any time soon.

Which is to say that I think poetry is likely stuck with this: and will just have to keep recommending that folk clear their cache when they hit it.
Thanks for the input and clarification, David. Your description makes total sense to me. I was in an airport with a very bad internet connection and probably hit Ctrl+C to restart the network or something similar. 

It's good that we found a possible culprit for this issue. Sadly it is not something that poetry/contributors can change/fix anytime soon, but it's 100% understandable why.

I'm happy with the solution clearing cache as it works perfectly. Having said that, I would close the issue. However, I do not know if you guys prefer to have it open to keep track of a (possible?) change in `cachecontrol`? I give control to you of the issue status 😆
Related to:

- #5922
> I don't know whether it's even possible to write filelock or equivalent in a way that doesn't sometimes leak an unwanted lockfile

Yes. Many Unixes have `flock()`, which gets automatically unlocked when the file descriptor is closed (e.g. when the process exits or is terminated for any reason). On Windows it's possible to set files to be deleted when their handle is closed.

> filelock is deprecated, but efforts to replace it in cachecontrol are not progressing

Nit: `cachecontrol` uses `lockfile`, not `filelock`. `lockfile` uses some weird hardlink API that requires explicit cleanup, but `filelock` uses `flock` internally, which as described above, gets automatically released on exit.


Resolving patch:
diff --git a/src/poetry/utils/authenticator.py b/src/poetry/utils/authenticator.py
--- a/src/poetry/utils/authenticator.py
+++ b/src/poetry/utils/authenticator.py
@@ -12,12 +12,14 @@
 from typing import TYPE_CHECKING
 from typing import Any
 
+import lockfile
 import requests
 import requests.auth
 import requests.exceptions
 
 from cachecontrol import CacheControl
 from cachecontrol.caches import FileCache
+from filelock import FileLock
 
 from poetry.config.config import Config
 from poetry.exceptions import PoetryException
@@ -33,6 +35,26 @@
 logger = logging.getLogger(__name__)
 
 
+class FileLockLockFile(lockfile.LockBase):  # type: ignore[misc]
+    # The default LockFile from the lockfile package as used by cachecontrol can remain
+    # locked if a process exits ungracefully.  See eg
+    # <https://github.com/python-poetry/poetry/issues/6030#issuecomment-1189383875>.
+    #
+    # FileLock from the filelock package does not have this problem, so we use that to
+    # construct something compatible with cachecontrol.
+    def __init__(
+        self, path: str, threaded: bool = True, timeout: float | None = None
+    ) -> None:
+        super().__init__(path, threaded, timeout)
+        self.file_lock = FileLock(self.lock_file)
+
+    def acquire(self, timeout: float | None = None) -> None:
+        self.file_lock.acquire(timeout=timeout)
+
+    def release(self) -> None:
+        self.file_lock.release()
+
+
 @dataclasses.dataclass(frozen=True)
 class RepositoryCertificateConfig:
     cert: Path | None = dataclasses.field(default=None)
@@ -122,7 +144,8 @@ def __init__(
                     self._config.repository_cache_directory
                     / (cache_id or "_default_cache")
                     / "_http"
-                )
+                ),
+                lock_class=FileLockLockFile,
             )
             if not disable_cache
             else None


Test patch: