pypa/pip — pull request #1812 by dstufft (2014-05-10 21:06:23+00:00)
Title: Condense --allow-external and --allow-unverified into one option.

The decision to support --allow-external and --allow-unverified was made in order to allow people to safely host their files externally to the index (generally PyPI). However since the implementation of this it has become abundantly clear that people are simply not choosing to host their files externally but safely. Additionally the difference between a external but safe and an external but unsafe file is dificult to explain and leaves users confused.

This will condense the two into a single option, --allow-external which has the semantics of both options specified together. This means that it is no longer possible to select safely externally hosted files but exclude unsafely externally hosted files.

This is spawned from [this distutils-sig thread](https://mail.python.org/pipermail/distutils-sig/2014-May/024200.html) (which itself was spawned from [this python-dev thread](https://mail.python.org/pipermail/python-dev/2014-May/134448.html)).

The below text is an adapted reproduction from one of my posts in the distutils-sig thread.

#### Rationale

I unequivocally believe that unsafe downloads *must* be opt in by default.

I also believe that external downloads *should* be opt in by default.

In the current situation we have two knobs that control these independently. The paranoid security person in me loves this because it means that for some set of projects I can still opt in to the reliability hit but not the security hit. However the UX person in me hates this because more knobs is more confusion, especially in this situation because the line between what is external+safe and what is external+unsafe isn’t very easy to explain.

So In my mind I've had to reconcile between these two viewpoints and when I look at the set of projects which are utilizing the external+safe hosting option I cannot find anything that tells me that many people are ever going utilize the external+safe option because the fact is projects simply are not using it in any meaningful numbers.

There are currently 0.06% (23 total) of projects on PyPI that have *all* of their files hosted off of PyPI but done so safely. Looking closer at them I can see that the number that have files that will actually be installed by pip specifically that number drops down to 0.04% (15).

Originally I had pointed out that 0.2% of projects host *any* files externally but safely. Looking again closer at them and removing projects which have also uploaded all files to PyPI, or which the file(s) that are safely hosted externally are not otherwise suitable I've determined that only 0.08% (32) of projects which I was able to discover any files for would be helped *in any way* by the external+safe option. And looking even closer at those, only 0.07% (26) of them will have the outcome of ``pip install whatever`` change (in other words, the latest version requires external+safe).

So when I look at the data, I cannot make a very good claim to the UX side of me that external+safe deserves it's own option when the number of projects which would ever use it instead of external+unsafe is minuscule, and of those projects I can only point to argparse and mysql-connector-python which are likely to affect many people at all.

So my beliefs are (in order of priority/conviction):

1. Unsafe downloads *MUST* but opt in.
2. External downloads *SHOULD* be opt in.
3. There is not enough potential users for separate knobs that allow external+safe or external+unsafe individually and they should be collapsed into a single option.

So following those beliefs lead me to conclude that the best result for these options are (in order of preference):

1. Collapse the two options into a single option and have it off by default.
- Satisfies 1 and 2 because they are opt in.
- Satisfies 3 because users don't have to futz with two different options.
- Slightly makes me sad that people can't install externally things safely, but the UX win makes up for it.
2. Leave the situation as, two options and one off by default.
- Satisfies 1 and 2 because they are opt in.
- Throws away 3.
3. Keep the options separate, but enable --allow-all-external by default and re-add the --no-allow-external option.
- Satisfies 1 because it is opt in.
- Makes 2 sad because it is opt out.
- Throws away 3
4. Remove the --allow-external family of options and enable them by default and always.
- Satisfies 1 because it is opt in.
- Throws away 2.
- Throws away 3.

There are bother benefits to option 1 of course. I'm someone of a public figure with packaging so people often come to me with questions, concerns, comments, etc. In that capacity I've had a number of people confused about the difference between an external file and an unverifiable file. In these cases I've had some difficulty in explaining what the difference is, especially since a lot of people have zero idea how the installer API even works. In the words of the Zen -> "If the implementation is hard to explain, it's a bad idea." and I can tell you that it is quite difficult to explain it to people. The rules are:

```
If there is a tag:
Trust all URLs with rel="internal" on the simple page
Require --allow-external for any URL on the simple page that links
directly to something that looks installable to pip and that also
includes a hash fragment like #= which can
be either md5, sha1, or in the sha-2 family.
Require --allow-unverifiable for any URL on that simple page that
directly links to something that looks installable to pip and that
does not include a hash fragment with a hash in it. Also any URL
that can be found by looking for URLs on the simple page that are
linked with a rel=download or rel=homepage, fetching that page,
and processing it's HTML looking for direct links to files that
look installable to pip.
else:
Trust all URLs that directly link to a file on the simple page
Trust all URLs that can be found by looking for links with rel=download
or rel=homepage, fetching that page, and processing it's HTML looking
for direct links that look installable to pip.
```

That's complex, and quite often when I explain that the first response is "what's a simple index?". Although I mostly only need to explain the first part and the "else" part I don't because most people don't install from not-PyPI.

On the flip side option (A) allows us to make this much simpler overall. We can simply do:

```
If it's hosted on PyPI:
Trust it.
else if it's not hosted on PyPI:
Require a --allow-external
```

This is *much*, *much*, *much* easier to explain, and I think it may be a good idea ala the Zen.