Repository: pypa/pip

Issue:
Un-deprecate `--process-dependency-links` until an alternative is implemented
* Pip version: 8.1.1 
* Python version: 3.5.2
* Operating System: Ubuntu 16.04

### Description:

The `--process-dependency-links` was re-added to pip a while ago because there are a number of valid use-cases for the flag: https://github.com/pypa/pip/pull/1955

For this reason, it should probably be un-deprecated until an implementation of PEP 508/440 is in pip. The `dependency_links` argument to setuptools's `setup` is also still documented and not deprecated: http://setuptools.readthedocs.io/en/latest/setuptools.html#dependencies-that-aren-t-in-pypi

### What I've run:

```
$ pip install request --process-dependency-links
Collecting request
  Downloading request-0.0.12.tar.gz
  DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
... 
```



Discussion:
Agreed. Current behaviour of pip for dependency-links really sucks.
#4295
#3610
[Wrote in #3939, moving my comment here]

> So, here's the main issue: If I *don't* want to use a `requirements.txt` file (because, say, I want declarative dependencies all specified in `setup.cfg`), how am I supposed to specify a URL as dependency?
> 
> This doesn't work:
> 
> ```ini
> [options]
> install_requires = 
>   requests==2.18.4
>   https://github.com/example/example_lib/archive/master.zip
> ```
> 
> I also think dependency links are weird and fine to drop, but *canonically* how is this use case served if not with those?
Another issue is that `dependency_links` apparently cannot be specified in setup.cfg, only as setup.py arguments. If they are to be undeprecated, that should be fixed.
Is there any news regarding this? Is there any alternative in progress? dependency_links_ seems to be the only option for internal/private distribution of libraries.

 Instead of deprecating it, it should be fixed, a lot of people forget or get confuse because they forget to put both the name-version of the library in the the install_requires and the dependency link that contains the tarbal together with the egg. As an alternative, I would love to see that @jleclanche has suggested, a simple link to the tarball in the install_requires (specifying the egg could be optional).

I don't think we are alone: https://stackoverflow.com/questions/3472430/how-can-i-make-setuptools-install-a-package-thats-not-on-pypi
@robertour tools like devpi support a more clear approach since years now
@dstufft btw - i would like to argue that the existene of devpi demonstrates a more clean solution to the problem making dependency links unnecessary

IMHO index inheritance +  white listing/blacklisting is more consistent and more comprehensible at a ops level
@RonnyPfannschmidt, thanks for the suggestion, it seems I am missing something important (discussions about this deprecation have been going for too long) but generally speaking I cannot agree more with this comment from [reddit](https://www.reddit.com/r/Python/comments/27jdba/dependency_links_will_go_how_to_deal_with_nonpypi/): 

> devpi could be a solution, but bringing another service into the game that needs maintenance and having to wait for some administrator to setup, will not get me into working very soon. Also we still would have the need to build packages and then put them onto devip (or has devpi a feature to track git repositories??)

Ideally, I would like an alternative that won't take me more than adding a file in my library, or few lines in the setup.py.

I basically don't understand the underlying decision of removing it, I can see how it can lead to bad practices and insecurities, but as long as it is keep for internal use, it should not be a major issue. 

#4175 means that PEP 508 URL support will be there in pip 10. I think this can be closed.

Thoughts @pfmoore @dstufft?
I thought https://github.com/pypa/pip/pull/4175/commits/86b07792e7ae762beeaeb471ab9db87e31bc285d meant that the `@` syntax cannot be used for dependencies, so this means #4175 isn't a solution for this issue. The comments over there were that we shouldn't implement `@` support for dependencies until PyPI was able to block uploads that used it (because we don't want installing something from PyPI to be able to grab stuff from arbitrary URLs, presumably).
It shouldn't be deprecated until there is an alternative in place, i.e. PEP 508 support.  Until then, too many people are using it.  
What's the proper workflow for this?

Let 's say there's a library that I need to add a feature to. I fork it on Github. Feature isn't going to be merged any time soon, so I have a tool that I set to use my Github fork instead of PyPi.

Now all of my users have to add --process-dependency-links when installing my tool with pip, which is a deprecated and even once removed flag?

Is there some option in setup.py that I'm missing or is there really no way around this? It seems like the only viable way to do this is to push up a forked pypi package. That is just going to add to user's confusion once your pull request does get merged, anyway.
Let's come around to this.

`--process-dependency-links` does not really have an alternative today. So, I'll suggest that we go ahead and un-deprecate it.

However, the flag does mean that pip would reach out to arbitrary URLs -- it should have a warning attached to it about this.

@pradyunsg unless there is an actual plan to kill it off for good, i strongly suggest to let that zombie rest or be removed

i dont think the people that actually need it have any incentive to fix the situation unless pip says "NO i dont take on your debt"
As a pip maintainer, my concern is not allowing pip to be a vector for exploits. Pip assumes PyPI as the default index, so there's implicit trust in PyPI there. So my first questions are:

1. Do packages exist on PyPI that have `dependency_links` metadata?
2. Could PyPI be modified to refuse upload of projects with `dependency_links` (or does it do that already)?
3. Alternatively, could pip simply refuse to process dependency links (regardless of whether the option is set) for packages downloaded from PyPI?

If we could be sure that there was no way for pip users to be bitten by packages on PyPI with malicious `dependency_links`, I'd be less worried. Users opting to use a custom index have to decide for themselves whether to trust it. (I'd still prefer to retain the flag even then - this is potentially a remote exploit and should *always* be explicitly opt-in).

So, with the following changes:

1. Pip will never process dependency links from PyPI (either explicitly, or just because we know they aren't there)
2. The `--process-dependency-links` flag is un-deprecated, but only works for custom indexes.

I'd be OK with accepting that as the status quo. People needing a dependency link style feature can then thrash out amongst themselves how (if at all) they want to move forward.

If someone wanted to, we could put all of that in a standard defining dependency link metadata, and the current deprecation would then become "the current behaviour will be dropped in favour of the behaviour specified in the standard". But from a pip-centric point of view, I'd rather just do it - let the people who need dependency links take on the work of standardising (it's to their benefit after all, as they then have a route - getting a standard change - to modifying pip's behaviour with full community consensus).
Basically we have a feature that is meant to replace dependency links, but pip hasn't started respecting it yet, because we don't yet have a way to block uploads to PyPI that use it.
Do you mean #4175? Which is blocked because of 86b0779? Maybe we could modify https://github.com/pypa/pip/blob/master/src/pip/_internal/req/req_install.py#L167 to only give the error if `comes_from` is PyPI?

Then we could change the deprecation warning for `--process-dependency-links` to say that users should switch to the `@` syntax, and drop `--process-dependency-links` after a reasonable deprecation period (we'd have to start the clock ticking from the point where `@` syntax becomes available).
@pfmoore Yea that is exactly it, and that seems like a possible path forward.
OK. If no-one else gets to it first, I'll see about writing a PR.
From what I gather, the user issue here is that they need to pass an option that explicitly says "hey, I'm deprecated" when you pass it for functionality that doesn't have an alternative.

I'm in favor of dropping dependency_links support entirely over a deprecation period. And in the mean time, adding PEP 508 URL dependency support, which I do think pip should still be loud about when used and that they should still be opt-in. 

That would mean:

- put the ability to process PEP 508 url-deps behind a flag
  - aside: should name the flag such that a security concerned user would check the docs
- start refusing to install PEP 508 url-deps when they come from a PyPI package
- dependency-links stay deprecated but we have a date to we'll drop them
  - honestly, I don't want to give this any longer a deprecation period than a single release printing the alternative.

I typed that yesterday but didn't send it. I now realize that it's basically the same thing as @pfmoore's last comment.

<sub>yay! :)</sub>

Should URL deps require an opt-in flag? PEP 508 doesn't say so, and the current implementation doesn't do that. I can see the logic, but I don't trust my judgement on security vs ease of use questions.

Also, I'd like to see some clear documentation on how projects should switch from dependency links to @ syntax before we start pushing the switch. It's hard enough already to reach people who will be affected by deprecations, and warnings that they can't work out how to address won't help. Ideally, the warning should include a pointer to a "how to change over" document, IMO.
> I don't trust my judgement on security vs ease of use questions.

I'm inclined toward: "be secure by default" and opt-in for less secure behavior.

> clear documentation on how projects should switch from dependency links to @ syntax before we start pushing the switch

Sounds good to me.

> I'm inclined toward: "be secure by default" and opt-in for less secure behavior.

As someone who sits behind hard to deal with firewall and security constraints at my work, I'm very aware that we don't have much understanding of the sort of environments where this feature is likely to be most useful (internal development, closed source, tightly controlled workflows and network layouts, etc). So my inclination is to be secure for the default workflow (PyPI) but not put additional roadblocks in the way of people with use cases that are clearly important to their workflow, but where we don't properly understand the trade-offs they have to make.

To my mind, "if the package is coming from a server that you explicitly opted into using" is sufficient opt in to allow URL links.

See? It's not that I don't *have* opinions, just that I'm not sure my biases aren't showing :smile: 
> not put additional roadblocks in the way of people with use cases that are clearly important to their workflow, but where we don't properly understand the trade-offs they have to make.

Fair enough. If some users could provide insights into their workflow, that would be nice.

> "if the package is coming from a server that you explicitly opted into using" is sufficient opt in to allow URL links.

I'm not sure here. :/

> See? It's not that I don't have opinions, just that I'm not sure my biases aren't showing 😄

Hehe.

> Fair enough. If some users could provide insights into their workflow, that would be nice.

> To my mind, "if the package is coming from a server that you explicitly opted into using" is sufficient opt in to allow URL links.

Just allowing sources from GitHub would solve 99% of the issue for me.  Upstream packages have bugs or missing features.  I fork and fix them, issue a PR, then wait for a possibly really long time to get them merged and then released on PyPI.  In the meantime my packages rely on the fixes in those packages.  

This can be opt-in, as long as it can be done in one line.  
It's not really a security issue to use direct links (assuming HTTPS/etc), it's just an availability one. Since we're disallowing them from PyPI then I'd say that's good enough and we don't need another flag.
At my work we would fall in some of the use cases @pfmoore mentioned, namely, internal development before a package is open sourced, closed source, etc. (always internal packages depending on other internal packages, all of them in a version control server).
Although, I also see the possible use case of someone referring to a file system location... 
Would it make sense to provide a list of whitelisted hosts/locations? The name of the flag should IMO, as @pradyunsg suggested, be descriptive enough to make the user aware of the risks involved. Maybe `--whitelisted-host`?
@masdeseiscaracteres I think you may have misunderstood - I was suggesting that if a package was retrieved from PyPI, we would fail if any of its dependencies were specified via an `@` reference. But there shouldn't be any such cases on PyPI anyway (we're expecting to reject them at some point, we just haven't been able to set that up yet). All other uses of `@` references would be OK.
Looks like we're going to go ahead with a PR that does what's described by @pfmoore in https://github.com/pypa/pip/issues/4187#issuecomment-389846189.

PRs welcome. :)
Note that I haven't had time to get to this - not because the change is difficult (as noted, it appears to simply be a check against `comes_from`) but rather because I don't know how to provoke the error myself (and more important, I don't know how to write a test that does so). If anyone can provide an example of such a test case that would be immensely helpful.
> If anyone can provide an example of such a test case that would be immensely helpful.

There's an existing test `test_install_pep508_with_url_in_install_requires` that demonstrates this.

As for erroring out when installing from PyPI, I don't see a better option than actually uploading something on PyPI that has a URL requirement. 😞 I uploaded a package on PyPI for this purpose. https://pypi.org/project/pep-508-url-deps/

---

Another thing is -- `comes_from` is not a URL or path, it's a string along the lines of `'box==0.1.0 from file:///private/tmp/box'`. Whoever's looking to fix this issue, now has to figure out a better way to error out, so that we have information about where a package originates. :)

@pfmoore this issue is near and dear to my heart 😄  Does @pradyunsg 's upload give you what you need, and are you still planning to tackle this? If not, I could take a swing at it.
@bstrdsmkr Nope, and as @pradyunsg says, it's not as simple as I thought, as `comes_from` isn't the source URL. So I don't know when I'll have time now (I have no personal use for this feature, so it's not high on my priority list).

As noted above, PRs welcome :-)
I'll add that the uploaded package doesn't help in any way for implementation, it's only helpful for testing the functionality.
My understanding is that the desired fix is something like:
```python
if req.url and is_like(req.url, PYPI.url):
    raise
```
in https://github.com/pypa/pip/blob/master/src/pip/_internal/req/req_install.py#L172
where `is_like()` returns `True` if the urls root at the same domain. Is that correct?
Yeah. I think so. That'll be the code change.

And, we'll need to add/update tests and a NEWS entry.

I also think that this change is important enough to warrant a chance the deprecation message providing link to docs + an additional section in user-guide telling people how to switch to the alternative.
>As a pip maintainer, my concern is not allowing pip to be a vector for exploits. Pip assumes PyPI as the default index, so there's implicit trust in PyPI there.


No, there is explicit trust. And adding safeguards against using external sources in dependencies IMHO won't improve the situation: it's more convenient to hide malware in pypi than in a publicly available VCS.

IMHO a better approach would be to use developer's VCSs as primary sources and keep pypy as a registry pointing to them and a caching proxy with some strong  cryptographical proof that the content in VCS is identical to the one got from pypy. I mean 

0 registration
```
dev -- public key --> pypa
```

1 uploading
```
setuptools -- git+https:/.... --> pypa
pypi --> Tor --> give me that commit --> vcs
pypi <-- Tor <-- here -- vcs
pypi checks the signature matches the dev
```
2 idle:
```
pypi --> Tor --> give me that commit --> vcs
pypi <-- Tor <-- here -- vcs
pypi checks it
```

2 retrieving
```
flips a coin
if coin == 1:
  pip -- give me package git+https:/... --> pypi
  pip <-- signature || content -- pypi
  pip -- give me the signature --> vcs
  pip <-- signature -- vcs
else:
  pip -- give signature of package git+https:/... --> pypi
  pip <- signature -- pypi
  pip --> Tor --> give me that commit --> vcs
  pip <-- Tor <-- here -- vcs


pip checks if the signature matches the public key and signature from pypa
pip -- give me public key --> keyserver
pip <-- PK -- keyserver
pip checks signature given by VCS against the sdist given by pypy
pip caches public key and repo location
```

1 the sources installed match the ones in VCS because of signature
2 the author matches too
3 pypa can cheat with new users, but cheating may be detected by old ones
4 author can cheat by showing in web interface branch other than given to pypy and send to pypy another branch. It won't work well if we make the branch a mandatory part of URI.
@KOLANICH I think you mean PyPI (Python Packaging Index) when you say pypy/pypa. PyPy is an alternative implementation of Python. PyPA (Python Packaging Authority) is a group of volunteers who maintain major projects in the Python Packaging space. Please, do get the acronyms correct or refrain from using them.

You're suggesting fundamentally changing the design of an existing well established service. If you wish to have such major changes, you're welcome to provide at least a (feasible) transition plan and a POC implementation, for managing/changing the architecture to allow transition/minimize breakage of existing workflows. Note that depending on external hosting is something that was explicitly removed from PyPI in the past in [PEP 470](https://www.python.org/dev/peps/pep-0470/), however that was fairly different case from what you're suggesting.

PyPI is maintained by volunteers, running on donated/sponsored infrastructure. You're suggesting that it connect through Tor, another volunteer maintained/run service. Both of these are major projects and they have a cost associated with keeping them running, even though it is not directly bourne by/visible to it's users.

---

None the less, this is not the right place for this discussion. If you do wish to propose a redesign of PyPI, I suggest you get a discussion started over at https://github.com/pypa/warehouse.
Thank you for the suggestions.

Resolving patch:
diff --git a/src/pip/_internal/models/index.py b/src/pip/_internal/models/index.py
--- a/src/pip/_internal/models/index.py
+++ b/src/pip/_internal/models/index.py
@@ -2,14 +2,20 @@
 
 
 class Index(object):
-    def __init__(self, url):
+    def __init__(self, url, file_storage_domain):
         self.url = url
         self.netloc = urllib_parse.urlsplit(url).netloc
         self.simple_url = self.url_to_path('simple')
         self.pypi_url = self.url_to_path('pypi')
 
+        # This is part of a temporary hack used to block installs of PyPI
+        # packages which depend on external urls only necessary until PyPI can
+        # block such packages themselves
+        self.file_storage_domain = file_storage_domain
+
     def url_to_path(self, path):
         return urllib_parse.urljoin(self.url, path)
 
 
-PyPI = Index('https://pypi.org/')
+PyPI = Index('https://pypi.org/', 'files.pythonhosted.org')
+TestPyPI = Index('https://test.pypi.org/', 'test-files.pythonhosted.org')
diff --git a/src/pip/_internal/req/req_install.py b/src/pip/_internal/req/req_install.py
--- a/src/pip/_internal/req/req_install.py
+++ b/src/pip/_internal/req/req_install.py
@@ -32,6 +32,7 @@
 from pip._internal.locations import (
     PIP_DELETE_MARKER_FILENAME, running_under_virtualenv,
 )
+from pip._internal.models.index import PyPI, TestPyPI
 from pip._internal.req.req_uninstall import UninstallPathSet
 from pip._internal.utils.deprecation import (
     RemovedInPip11Warning, RemovedInPip12Warning,
@@ -169,11 +170,19 @@ def from_req(cls, req, comes_from=None, isolated=False, wheel_cache=None):
             req = Requirement(req)
         except InvalidRequirement:
             raise InstallationError("Invalid requirement: '%s'" % req)
-        if req.url:
+
+        domains_not_allowed = [
+            PyPI.file_storage_domain,
+            TestPyPI.file_storage_domain,
+        ]
+        if req.url and comes_from.link.netloc in domains_not_allowed:
+            # Explicitly disallow pypi packages that depend on external urls
             raise InstallationError(
-                "Direct url requirement (like %s) are not allowed for "
-                "dependencies" % req
+                "Packages installed from PyPI cannot depend on packages "
+                "which are not also hosted on PyPI.\n"
+                "%s depends on %s " % (comes_from.name, req)
             )
+
         return cls(req, comes_from, isolated=isolated, wheel_cache=wheel_cache)
 
     @classmethod


Test patch:
diff --git a/tests/functional/test_install.py b/tests/functional/test_install.py
--- a/tests/functional/test_install.py
+++ b/tests/functional/test_install.py
@@ -8,6 +8,7 @@
 import pytest
 
 from pip._internal import pep425tags
+from pip._internal.models.index import PyPI, TestPyPI
 from pip._internal.status_codes import ERROR
 from pip._internal.utils.misc import rmtree
 from tests.lib import (
@@ -1267,9 +1268,31 @@ def test_install_pep508_with_url_in_install_requires(script):
             'ce1a869fe039fbf7e217df36c4653d1dbe657778b2d41709593a0003584405f4'
         ],
     )
-    res = script.pip('install', pkga_path, expect_error=True)
-    assert "Direct url requirement " in res.stderr, str(res)
-    assert "are not allowed for dependencies" in res.stderr, str(res)
+    res = script.pip('install', pkga_path)
+    assert "Successfully installed packaging-15.3" in str(res), str(res)
+
+
+@pytest.mark.network
+@pytest.mark.parametrize('index', (PyPI.simple_url, TestPyPI.simple_url))
+def test_install_from_test_pypi_with_ext_url_dep_is_blocked(script, index):
+    res = script.pip(
+        'install',
+        '--index-url',
+        index,
+        'pep-508-url-deps',
+        expect_error=True,
+    )
+    error_message = (
+        "Packages installed from PyPI cannot depend on packages "
+        "which are not also hosted on PyPI."
+    )
+    error_cause = (
+        "pep-508-url-deps depends on sampleproject@ "
+        "https://github.com/pypa/sampleproject/archive/master.zip"
+    )
+    assert res.returncode == 1
+    assert error_message in res.stderr, str(res)
+    assert error_cause in res.stderr, str(res)
 
 
 def test_installing_scripts_outside_path_prints_warning(script):