Repository: pypa/pip

Issue:
`vendoring` is broken, due to a cyclic dependency during license fetching
Well, the lack of maintainance of the license fetching logic in `vendoring` has come to bite us. :)

`flit` recently established a cyclic dependency, by depending on `tomli`: see https://github.com/takluyver/flit/issues/451 and https://flit.readthedocs.io/en/latest/bootstrap.html.

We get licenses from sdists in `vendoring` (which means building metadata / wheels -- leading to https://github.com/pradyunsg/vendoring/issues/1). Since flit is no longer bootstrappable through regular mechanisms, it'd be best to switch to using wheels for the license fetch phase.

This is the tracking issue for actually fixing this, and adopting the fix in our workflow.




Discussion:
Sorry about that breakage, and if you need me to do anything temporarily in Flit to keep things ticking over, let me know.
Nothing that comes to mind right now.

This is the sort of thing where doing a major version bump is controversial since SemVer doesn't really cover that -- it's a distribution-related change, not a usage change. 🙃

FWIW, why was this change made? I can't seem to find any discussion about it, and IIRC, being able to bootstrap backends was explicitly why backend-path was added in.

The change was switching the TOML parser in Flit to tomli - because that's the one everyone else is now using, and because it supports the TOML 1.0 standard (which the `toml` package didn't). `flit_core` can still bootstrap itself (it can build a wheel of itself without reading any TOML). But it needs the toml parser to package anything else - including tomli itself. It works if you can use tomli (which is pure Python) installed from a pre-built wheel, so I didn't think it would be too disruptive. :shrug: 

Previously we avoided this simply by making bootstrapping someone else's problem - toml was built with setuptools. But if you want to use PEP 517 all the way, that's actually no better: there's a cycle between setuptools & wheel. And I obviously like the idea of being able to bootstrap my packaging tool without setuptools. :slightly_smiling_face: 

IMO, the long term fix is putting a toml parser in the stdlib. But even if that's agreed now, it's going to be quite some time before we can actually rely on that.
Using a vendored Tomli in flit_core would be a way to break the cycle temporarily, until a TOML parser is available in the stdlib. But I don't how much @takluyver and some repackagers would hate doing that? I assume a lot. :smile: 
I don't particularly mind doing that - but I'd guess that there's a large overlap between the people who can't use a pre-built wheel and the people who reject bundled packages, so I don't know if bundling would help many people.
I'd be interested in understanding whether `backend-path` is actually useful here, and if not, why not (if "we still have to vendor stuff and that sucks" is the answer, then that's valid). Because as far as I recall, it was added explicitly to handle this type of situation, so if it's not doing that job, we maybe need to review it.
`backend-path` is working correctly for `flit_core`, as far as I'm concerned. It's [set here](https://github.com/takluyver/flit/blob/96751efce651f8bae8ccb9e7f144dac460b3f013/flit_core/pyproject.toml#L4), and flit_core can be built from source with no problem:

```
pip wheel --no-binary :all: --no-deps flit_core
```

The issue arises because flit_core has a *runtime* dependency (not a build dependency) on tomli. So tomli's build dependency on flit_core means it has a transitive build dependency on itself. Building flit_core is fine, using it for tomli is what falls down.

Another possible workaround would be to have a special build backend in *tomli* sdists - e.g. bundling flit_core in the sdists -  and using `backend-path` in tomli's pyproject.toml to use that. That would work, as far as I know, but I don't think it should be on tomli to do special workarounds for bootstrapping - if we need special cases, they should be in flit_core.

Yet another possibility is that pip could detect such a cycle - we're trying to build tomli, and we've found that we need tomli - and use a wheel in an isolated build env to break it, unless you pass some extra option `--no-wheels-not-even-to-break-build-cycles`. But that's probably a lot of work for a hopefully very specific situation.
Ah, right. backend-path solves the build time dependency problem, not the runtime one. :)

Resolving patch:
diff --git a/noxfile.py b/noxfile.py
--- a/noxfile.py
+++ b/noxfile.py
@@ -171,7 +171,7 @@ def lint(session: nox.Session) -> None:
 
 @nox.session
 def vendoring(session: nox.Session) -> None:
-    session.install("vendoring~=1.0.0")
+    session.install("vendoring~=1.2.0")
 
     if "--upgrade" not in session.posargs:
         session.run("vendoring", "sync", "-v")


Test patch: