Repository: jazzband/pip-tools (Python)

Issue:
pip-compile absolute links when given relative path
requirements.in:

```
-e .
-e ./sqlalchemy_jsonapi
```

compiles to:

```
-e file:///Users/coltonprovias/MyProject
-e file:///Users/coltonprovias/MyProject/sqlalchemy_jsonapi
```

It should compile to

```
-e .
-e ./sqlalchemy_jsonapi
```

It's not really useful for me to provide a requirements.txt file then to anybody unless they use a Mac and have the same name as me.  Although if that person does exist, I would like to meet them.



Discussion: I ran into the same issue today.  Why are relative paths are expanded?

This has affected me too as I'm trying to install packages which are submodules of my project. It seems the issues arises here: https://github.com/pypa/pip/blob/develop/pip/req/req_install.py#L1131

I suspect it is hard to avoid as `pip-tools` uses the `pip` library and the `pip` library has no reason to keep a relative path around. It wants to deal with absolute paths.

Which means that it is unlikely that we'd get the `pip` library changed to help us so which suggests that we'd have to get the output from the `pip` library and go back and make any paths relative again which is an option but not incredibly desirable as it might be hard to know exactly when to do it and when not to.

Personally, I'm going to try switching from using submodules and relative paths to having `git+git://` paths in my `requirements.in` file but at the same time it seems very reasonable to try to fix this as it makes little sense to have absolute paths in a `requirements.txt` file when there is little chance that the paths in different development environments or the production environment is going to be the same. That is certainly what affects me. Of course, the `requirements.txt` should also be in git so it should be possible to `pip-compile` and then edit the `requirements.txt` before committing it.

Looks like https://github.com/nvie/pip-tools/issues/325 is related

I've been planning to move all of the requirements handling across all of our repos to using pip-tools and just realized it had this issue, which may completely shelve that plan. This is a pretty significant problem.
So the core of this comes [down to this function here](https://github.com/pypa/pip/blob/2e534b0c923c96c6c3f7949e23df2573078cb9d7/pip/download.py#L470), inside `pip`. I don't think it would be easy to fix in `pip` itself, so I will see if I can manipulate the result while outputting it to the file by making the result relative.
the relative paths should actually use `file:foo/bar` so that the `#egg=spam` fragment could be appended. Unfortunately this breaks in `pip-compile`, as `pip-compile` tries reading `foo/bar/setup.py` *after* setting `foo/bar` as the current directory!
as for why `#egg=spam` would be needed, is for `pip` dependency resolver to find out what _abstract_ requirement is provided by the said file link! Otherwise it is going to attempt to download `spam` from Warehouse!
If your parrot is a git submodule, this works well: `-e git+path/to/dead/parrot#egg=parrot`.

@rollcat *almost* works... except that we merged 2 repos so it is a slug and hardly speaks.
I took another stab at trying to fix this in https://github.com/jazzband/pip-tools/pull/702.
I've been using the following post-processing step to get around the problem: `make_paths_relative.py < requirements.txt` (script below)

```python
#!/usr/bin/env python
import os
import re
import sys

pattern = re.compile(r"\s*-e\s+file://(?P<path>.*?)\n")
inputs = sys.stdin.readlines()
for line in inputs:
    match = pattern.match(line)
    if match:
        path = match.group("path")
        path = os.path.relpath(path)
        line = f"-e {path}\n"
    sys.stdout.write(line)
```
A(nother) hacky solution, in `requirements.in`:
```
-e file:relative/path/to/my_package
```
then to compile:
```
pip-compile -q -o - | sed s:///::g > requirements.txt
```
Ran into this issue as well. If I modify all my `-e src/foo` links to `-e file:src/foo#egg=foo` then the compiled requirements file drops out the same `-e file:src/foo#egg=foo` and this works as expected. Quite a lot more verbose than necessary though. The egg on the end is required, or the sed script above in https://github.com/jazzband/pip-tools/issues/204#issuecomment-486242541 otherwise the later `pip install -r` will not properly find the "foo" project to satisfy the dependency lists of other projects depending on "foo".
@mmerickel

> Ran into this issue as well. If I modify all my `-e src/foo` links to `-e file:src/foo#egg=foo` then the compiled requirements file drops out the same `-e file:src/foo#egg=foo` and this works as expected.

That's a nice workaround!
I don't see it mentioned specifically here, but I've had the same issue with `--find-links ./local_src`.  I want pip to look for packages within a directory as part of the project, but `pip-compile` translates that into an absolute path with './local_src' tacked on to the end.
@tisdall you could try `--find-links file:./local_src`. It keeps the relative path.
@atugushev thanks!  Strangely when I tried it I had both relative and the original full path until I completely deleted the 'requirements.txt' and then recompiled.
@tisdall looks like a bug. Could you file an issue?
+1 We are also dealing with this issue
I've pinned the issue to bring more attention to the most wanted feature based on reactions.
I'm happy to work on this and other path-sanity tasks, but would like some discussion re: #1067 before diving in.
Just ran into this issue today when migrating to pip-tools while still listing requirements inside setup.py, blocking our implementation.
I was trying to figure out the proper syntaxes supported for paths, by reading [here](https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format).

That indicates either of the following are acceptable:
```
[-e] [<vcs>+]file://<path>[#egg=<name>]
[-e] <name> @ file://<path>
```
Notes on that:
- Be careful, if you accidentally add a slash to a relative path (`file:///.`) you'll be trying to install a whole lot more than you bargained for!
- It seems relative paths here aren't supported by `file://`; if I have `file://.#egg=pip-tools` and try `pip install -r requirements.in`, it'll fail with:
```
ValueError: non-local file URIs are not supported on this platform: 'file://.'
```
- Using the undocumented (at least at the above link) syntax `file:.` will work with the first syntax, but not the second.
- Using `git+file:.` will not work.
- `git+file://` will not work with the second syntax.

So my understanding is now that these are the appropriate syntaxes for local paths:

```
[-e] [<vcs>+]file://<absolute>[#egg=<name>]
[-e] <name> @ file://<absolute>
[-e] file:<relative>[#egg=<name>]
```
Q1: Is this an acceptable standard for this project?

I haven't begun to look at the `setup.py` syntaxes.

Q2: Setting aside `setup.py` for the moment, if we abide by the above potential standard, what *doesn't* work?

Q3: I also want to note and check for agreement here that we should ensure the relative paths output are reformulated to be relative to the output file, if output "file" is not stdout.


Any news regarding this issue. Also ran into this "problem". Cheers.
Hello @zuckerruebe,

Thanks for pinging! As far as I know, nobody works on this issue at the moment. In the meantime, you might use these workarounds:
1. https://github.com/jazzband/pip-tools/issues/204#issuecomment-564597799
1. https://github.com/jazzband/pip-tools/issues/204#issuecomment-550051424
@zuckerruebe @atugushev and anyone else, can you provide any feedback for Q1, Q2, Q3 at the end of [my last comment](https://github.com/jazzband/pip-tools/issues/204#issuecomment-615949987) (jump to "So my understanding" above the last code block)?
> Q1: Is this an acceptable standard for this project?

IMO Instead
```
[-e] file:<relative>[#egg=<name>]
```
it should be:
```
[-e] <relative>[#egg=<name>]
```

AFAIK `file:./relative/path` is not a standard (more like a workaround) and we shouldn't rely on it in a long prospect future. I swear I saw a discussion about it on `pip` tracker, but can't find anymore :(

> Q2: Setting aside `setup.py` for the moment, if we abide by the above potential standard, what _doesn't_ work?

I think I'm not following the question here. Could you elaborate?

> Q3: I also want to note and check for agreement here that we should ensure the relative paths output are reformulated to be relative to the output file, if output "file" is not stdout.

Yes, makes sense to me.


**UPDATE:** found a discussion about `file:./relative/path` https://github.com/pypa/packaging/issues/120


Thanks @atugushev! 

---

### Q2

I'm asking: if users use one of the following syntaxes:

```
[-e] [<vcs>+]file://<absolute>[#egg=<name>]
[-e] <name> @ file://<absolute>
[-e] file:<relative>[#egg=<name>]
```

what exactly goes wrong, as the most basic case of the issue (compiling relpaths to relpaths) doesn't seem to be a problem? I notice two things to start with:

- only compiles if done from the same directory as the in-file
- the relpath is preserved literally, anchored to the path of the in-file, instead of the out-file

---

### Q1

After reading the pypa/packaging issue you linked (and spidering out from there), it looks like we're waiting for some things to shake out at pypa/pip and pypa/packaging:

- this issue will not be thoroughly resolved until at least:
  - pypa/pip#6658
  - pypa/packaging#120
- we should also be interested to track:
  - pypa/pip#7650
  - pypa/packaging#264
  - [PEP 610](https://www.python.org/dev/peps/pep-0610/)
  - pypa/pip#5204
  - pip's [test_basic_install_relative_directory](https://github.com/pypa/pip/blob/master/tests/functional/test_install.py#L436)
- you are right, `file:./relpath` is an unreliable non-[standard](https://tools.ietf.org/html/rfc3986#section-3.3), not to be encouraged (but it *is* tested for in pip's `test_basic_install_relative_directory`)
- but `[-e] <relative>[#egg=<name>]` is not great either. From [pip#6658 (comment)](https://github.com/pypa/pip/issues/6658#issuecomment-562973835): 
  > [simply `./some/path`] wouldn't be acceptable per PEP 508, so it would be hard to justify supporting much less get it working across all tools.
- I think we're waiting on better support for `name @ <relpath>`, and generally stronger consensus on proper spec -- relative paths are currently resolved based on different origins depending on tool/codepath...

We *may* *eventually* be able to focus on:

```
[-e] <name> @ [git+file://]<relative-or-absolute>[@<tag-ish>]
```
I just want to reiterate for all the watchers that while we're still waiting on path syntaxes to settle upstream, much of the time, for now, you'll get what you want by using `file:./relpath`, such as `-e file:.`.
I welcome any and all testers to try #1329, and please let me know if you consider this issue fixed by its changes. 
I also invite all interested users to provide insight on the best possible handling of relative vs absolute find-links paths, over at #453. Thanks for any thoughts and real world use cases!
Just for the record, the `-e` workaround mentioned above won't work for non-setuptools libraries. However, editable installs are being standardized (see PEP 660 and PEP 662), so hopefully this will be less of an issue in the near future.
Please have a go and let me know if you have any cases which are *not* solved by ~#1329~ #1650. Thanks!
@AndydeCleyre I know nothing about cases _not_ solved by #1650 . I tested it with two local relative paths involving 2 packages depending between them, both featuring quite complex legacy setup.py files, in either editable and not editable mode and I have faced no issues so far.

Is there any list of edge scenarios you would like to see tested? I'd love to see this in a public pip-tools release.
Thanks @n1ngu !

No particular edge cases that I can think of, the PR is just waiting around for a proper review, really.
A minor caveat is that your fork is lacking most tags, causing setuptools_scm to be helpless when trying to infer the package version, so I am having a hard time trying to install your development branch alongside pip-compile-multi. It would be awesome if you could push the 6.8.0 tag on the 1e05d0028ce16d4b2de53ae38a33df7feb84e800 commit! :pray: 
I just pushed the existing tags, if that helps.
I'm sorry say that #1650 has been rejected, and I encourage anyone interested in the problem to think about alternative approaches.
The workaround `-e file:local/path/to/lib#egg=lib` mentioned by @AndydeCleyre [here](https://github.com/jazzband/pip-tools/issues/204#issuecomment-785420275) works when using `requirements.in` (and will get copied to the final requirements.txt), but is there a working syntax if you want to use `setup.cfg` or `pyproject.toml` to define the requirements?
The `-e file:src/foo#egg=foo` work-around does not seem to work with extra requires, e.g. `-e file:src/foo[extra]#egg=foo`

```
pip._internal.exceptions.InstallationError: File "setup.py" not found for legacy project foo from file:///src/foo[extra]#egg=foo
```

I'll note here that, without much guarantee, as long as it's pretty simple to do, I'll try to keep my relpaths branch updated. You can use `pipx` to invoke it in a one-off fashion, like

```console
$ pipx run --spec 'git+https://github.com/AndydeCleyre/pip-tools@feature/relpaths-post-6.8.0' pip-compile
```

Doing so, you should be able to use the simple syntax:

```ini
-e ./src/foo[extra]
```
> The workaround `-e file:local/path/to/lib#egg=lib` mentioned by @AndydeCleyre [here](https://github.com/jazzband/pip-tools/issues/204#issuecomment-785420275) works when using `requirements.in` (and will get copied to the final requirements.txt), but is there a working syntax if you want to use `setup.cfg` or `pyproject.toml` to define the requirements?

Hi! Thanks for this great tool. I'm stuck on using pip-tools with pyproject.toml since I have extras that include other extras, wanted to see if there are any updates on this.
Do any pyproject.toml build backends support relative paths for dependencies? I don't _think_ Poetry or Flit do.

If my branch isn't working for certain extras configurations when using requirements.in please let me know, I'll try to reproduce and fix.
> Do any pyproject.toml build backends support relative paths for dependencies? I don't _think_ Poetry or Flit do.
> 
> If my branch isn't working for certain extras configurations when using requirements.in please let me know, I'll try to reproduce and fix.

The problem is not relative paths for me as it is that extra dependency groups that include other optional dependencies will be compiled to something with absotule paths.

Example:

```toml
# package_name is the name of the package itself
[project.optional-dependencies]
dev = ["package_name[test,tools]"]
tools = ["pip-tools"]
test = ["pytest""]
```
The generated pip-compile for `dev` extra will be something like this:
```requirements
package_name @ file:///path/to/package/dir
pip-tools
pytest
```

Which is nonsensical to be honset.
It looks like that's as intended based on discussion at #1685, and maybe @q0w or @atugushev can speak to the rationale.

I personally don't like specifying a package as depending on itself, because that seems like an impossible infinite loop, but I now know it's commonly done. Instead I maintain dependency groups as `*-requirements.in` files, and script injection of that data into the `pyproject.toml`.

So that example would be managed like:

`test-requirements.in`:
```
pytest
```
`tools-requirements.in`:
```
pip-tools
```
`dev-requirements.in`:
```
-r test-requirements.in
-c test-requirements.txt

-r tools-requirements.in
-c tools-requirements.txt
```
```console
$ pypc  # inject deps according to above files into pyproject.toml, part of zpy, my wrapper of pip-tools
$ grep -A 3 optional-dependencies pyproject.toml
```
```toml
[project.optional-dependencies]
dev = ["pip-tools", "pytest"]
tools = ["pip-tools"]
test = ["pytest"]
```

> It looks like that's as intended based on discussion at #1685, and maybe @q0w or @atugushev can speak to the rationale.
> 
> I personally don't like specifying a package as depending on itself, because that seems like an impossible infinite loop, but I now know it's commonly done. Instead I maintain dependency groups as `*-requirements.in` files, and script injection of that data into the `pyproject.toml`.
> 
> So that example would be managed like:
> 
> `test-requirements.in`:
> 
> ```
> pytest
> ```
> 
> `tools-requirements.in`:
> 
> ```
> pip-tools
> ```
> 
> `dev-requirements.in`:
> 
> ```
> -r test-requirements.in
> -c test-requirements.txt
> 
> -r tools-requirements.in
> -c tools-requirements.txt
> ```
> 
> ```
> $ pypc  # inject deps according to above files into pyproject.toml, part of zpy, my wrapper of pip-tools
> $ grep -A 3 optional-dependencies pyproject.toml
> ```
> 
> ```toml
> [project.optional-dependencies]
> dev = ["pip-tools", "pytest"]
> tools = ["pip-tools"]
> test = ["pytest"]
> ```



I was using a mix of `requirements.in` files and `pyproject.toml` + dynamic dependencies of `setuptools` to aleviate this issue, but setuptools does not support `-c` or `-r` flags in the dependency declaration files. I don't know what can be the best option here, the `pypc` package seems ancient and I would probably use a `sed` or `awk` based bash script to rewrite these lines, but that seems infinitely hacky anyway.

I was thinking of migrating to `setup.cfg` and using the `%option%` syntax. 
To be clear, the pypc here has nothing to do with the package of that name on PyPI, but is part of my project zpy (here on GitHub). You can check that implementation, but basically you can use tomlkit to inject the data without being hacky.
> I just want to reiterate for all the watchers that while we're still waiting on path syntaxes to settle upstream, much of the time, for now, you'll get what you want by using `file:./relpath`, such as `-e file:.`.

@AndydeCleyre - I'm super grateful for all the time, energy, and patience you've invested into this issue. My team would certainly still love to see this get supported at some point, but we also understand the challenges involved and the resistance to the changes at this point in time.

**QUESTION:** My question is related to the work-around. I was delighted to see that using `file:./package-name` in my `requirements.in` file worked (path remained relative in `requirements.txt`), but unfortunately, using `file:../package-name` did not (path became absolute, presumably due to the parent directory reference: `..`). Confusingly, using `-e file:../package-name` _did_ result in a relative in the lockfile. **Are you aware of a work-around that avoids `--editable` but still allows the resulting lockfile to use relative paths when it contains `..`?**

FWIW, my goal is to have a lockfile that I can use locally during development within a monorepo, using relative paths to express dependencies on packages elsewhere in the monorepo. I also want the same lockfile to be used in a multi-stage Dockerfile where we copy a virtual environment between stages, but nothing else. Since `--editable` installs don't actually keep the source code in the virtual environment, I don't think I can use them and still have a "portable" venv...

Any input or advice would be greatly appreciated!
@tboddyspargo 

I understand if this is not a suitable solution for your needs, but my `feature/relpaths-post-6.8.0` branch should indeed [work](https://github.com/jazzband/pip-tools/issues/204#issuecomment-1467330109) if you specify e.g. `../package-name` or `file:../package-name`.

It's been a while since I was testing out all the combinations of input and output observed with the `main` branch and don't know if there's a certain syntax that works there, but I wonder if a workaround for you might instead be found in using relpath symlinks to the packages you need.

```console
$ ln -s ../package-name
$ echo 'file:./package-name' >>requirements.in
```
> I wonder if a workaround for you might instead be found in using relpath symlinks to the packages you need.
> 
> ```
> $ ln -s ../package-name
> $ echo 'file:./package-name' >>requirements.in
> ```

Clever! That does seem to work! We'll consider using that as a short-term option. Many thanks!
> > Do any pyproject.toml build backends support relative paths for dependencies? I don't _think_ Poetry or Flit do.
> > If my branch isn't working for certain extras configurations when using requirements.in please let me know, I'll try to reproduce and fix.
> 
> The problem is not relative paths for me as it is that extra dependency groups that include other optional dependencies will be compiled to something with absotule paths.
> 
> Example:
> 
> ```toml
> # package_name is the name of the package itself
> [project.optional-dependencies]
> dev = ["package_name[test,tools]"]
> tools = ["pip-tools"]
> test = ["pytest""]
> ```
> 
> The generated pip-compile for `dev` extra will be something like this:
> 
> ```
> package_name @ file:///path/to/package/dir
> pip-tools
> pytest
> ```
> 
> Which is nonsensical to be honset.

It's because how pip resolves recursive extras. When you do `pip install .[dev]` it works only because `.` converts to LinkCandidate containing your local path. So without `@ filepath` it would be package from pypi
> > > Do any pyproject.toml build backends support relative paths for dependencies? I don't _think_ Poetry or Flit do.
> > > If my branch isn't working for certain extras configurations when using requirements.in please let me know, I'll try to reproduce and fix.
> > 
> > 
> > The problem is not relative paths for me as it is that extra dependency groups that include other optional dependencies will be compiled to something with absotule paths.
> > Example:
> > ```toml
> > # package_name is the name of the package itself
> > [project.optional-dependencies]
> > dev = ["package_name[test,tools]"]
> > tools = ["pip-tools"]
> > test = ["pytest""]
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > The generated pip-compile for `dev` extra will be something like this:
> > ```
> > package_name @ file:///path/to/package/dir
> > pip-tools
> > pytest
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > Which is nonsensical to be honset.
> 
> It's because how pip resolves recursive extras. When you do `pip install .[dev]` it works only because `.` converts to LinkCandidate containing your local path. So without `@ filepath` it would be package from pypi

Using `setup.cfg` and `%(extra)s` directives in declaring optional dependencies allievated the issue. Do not know why `pyproject.toml` does not support that btw.

```cfg

[options.extras_require]
test = 
   pytest

dev = 
   pip-tools

all =
   %(dev)s
   %(test)s
```
~~Did you provide setuptools as build backend in pyproject.toml?~~ Does not work
> Do any pyproject.toml build backends support relative paths for dependencies? I don't _think_ Poetry or Flit do.

Just to answer this question directly: Poetry _does_ support relative paths for dependencies, I've tried it with a real example in one of my repos.
I'm coming here from the https://github.com/jazzband/pip-tools/issues/325 tl;dr.

In [pylint-dev/pylint-pytest@`a410e15` (#33)](https://github.com/pylint-dev/pylint-pytest/pull/33/commits/a410e15fe81f5a1e50070c4d56491ad09c0916fe) I am seeing that `pylint` (a dependency of `pylint-pytest`) is disappearing (and hence my action fails).

I'm assuming that Dependabot does not pick up the two arguments to the pip-compile for some reason.

Trying to use the https://github.com/jazzband/pip-tools/issues/325#issuecomment-552212086 as a workaround (https://github.com/pylint-dev/pylint-pytest/commit/b51c82fdf4f44edf896c584a6277e01b8d799294), then I get https://github.com/pylint-dev/pylint-pytest/actions/runs/7688457033/job/20949696021?pr=33#step:8:23

```
ERROR: The editable requirement pylint-pytest from file:///.#egg=pylint-pytest (from -r /tmp/tmp985jkgq3 (line 96)) cannot be installed when requiring hashes, because there is no single file to hash.
```

I want to require hashes (ofc), but it might be hard to get hashes from `-e file:.#egg=foobar`. The suggestion `Consider using a hashable URL like https://github.com/jazzband/pip-tools/archive/SOMECOMMIT.zip` might be helpful for CI (eventually), but I'd like to solve the local environment problem too.

Any ideas? 😕 



Fix (patch):
diff --git a/piptools/_compat/pip_compat.py b/piptools/_compat/pip_compat.py
index 6409fbc2a..a5f843ad9 100644
--- a/piptools/_compat/pip_compat.py
+++ b/piptools/_compat/pip_compat.py
@@ -10,6 +10,7 @@
 from pip._internal.metadata import BaseDistribution
 from pip._internal.metadata.pkg_resources import Distribution as _PkgResourcesDist
 from pip._internal.models.direct_url import DirectUrl
+from pip._internal.models.link import Link
 from pip._internal.network.session import PipSession
 from pip._internal.req import InstallRequirement
 from pip._internal.req import parse_requirements as _parse_requirements
@@ -63,6 +64,14 @@ def _from_importlib(cls, dist: _ImportLibDist) -> Distribution:
         return cls(dist._dist.name, dist._dist.version, requires, dist.direct_url)
 
 
+class FileLink(Link):
+
+    @property
+    def file_path(self) -> str:
+        # overriding the actual property to bypass some validation
+        return self._url
+
+
 def parse_requirements(
     filename: str,
     session: PipSession,
@@ -74,7 +83,14 @@ def parse_requirements(
     for parsed_req in _parse_requirements(
         filename, session, finder=finder, options=options, constraint=constraint
     ):
-        yield install_req_from_parsed_requirement(parsed_req, isolated=isolated)
+        install_req = install_req_from_parsed_requirement(parsed_req, isolated=isolated)
+        if install_req.editable and not parsed_req.requirement.startswith("file://"):
+            # ``Link.url`` is what is saved to the output file
+            # we set the url directly to undo the transformation in pip's Link class
+            file_link = FileLink(install_req.link.url)
+            file_link._url = parsed_req.requirement
+            install_req.link = file_link
+        yield install_req
 
 
 def create_wheel_cache(cache_dir: str, format_control: str | None = None) -> WheelCache: