Downstream unbundlers should populate setup.py

/cc @ralphbean @eriol

OpenStack has encountered yet another breakage caused by people combining `pip` installed and system packages: specifically, updating urllib3 via `pip` when requests/urllib3 are already installed via the system packages.

@dstufft tells me that, if the system packages correctly populate the `install_requires` part of `setup.py`, `pip` would respect those requirements. Previously this wasn't possible because requests would include mid-release versions of urllib3, but as of v2.6.2 our [new release rules](http://docs.python-requests.org/en/latest/community/release-process/) mean that we only use proper urllib3 releases.

For this reason, I think it would be extremely good if downstream unbundlers (most notably Fedora and Debian), when they unbundle requests, would populate setup.py with the correct `install_requires`. For 2.8.0, that would be `urllib3==1.12`. For 2.7.0, that would be `urllib3==1.10.4`. This should ensure that `pip` won't casually install a version of urllib3 that will break the system requests package.

Thoughts?


I think we can do that.  I'll have to think about the implications but this is a great improvement.  Right now Debian has requests 2.7.0 and urllib3 1.11 so they should be updated.


>  Right now Debian has requests 2.7.0 and urllib3 1.11 so they should be updated.

I've heard of issues with 2.7.0 and 1.11. I think Fedora pinned to 1.10.4. I'm surprised you haven't had complaints yet.


:smiley: Thanks Barry. For the sake of keeping track of everything, it would be _super_ convenient if you could drop a link to any Debian discussion of that change in this issue. Obviously if you can't then that's not a big deal.


My plan is to update both soon! I was waiting for the DPMT migration to git. I don't see particular problems but we can always upload on experimental first to see if it's all right.


On Oct 09, 2015, at 11:09 AM, Daniele Tricoli wrote:

> I was waiting for the DPMT migration to git.

Done!

https://wiki.debian.org/Python/GitPackaging


@warsaw thanks to make this happen! :smile: 
I started reading debian-python@l.d.o after my reply here!
@Lukasa if you can point me to the OpenStack issue I will include it into the description of the patch Debian will use.
Thanks!


> Thoughts?

Good idea.  Will report back when its in place.


@eriol OpenStack issue is [here](https://review.openstack.org/#/c/213310/0).


Patch to Fedora .spec files:
- http://pkgs.fedoraproject.org/cgit/python-requests.git/commit/?id=f73b98e44a519ee15f3ceeea4c53e0e4dd25a5d4

Fedora updates created:
- https://bodhi.fedoraproject.org/updates/FEDORA-2015-20de3774f4
- https://bodhi.fedoraproject.org/updates/FEDORA-2015-1f580ccfa4
- https://bodhi.fedoraproject.org/updates/FEDORA-2015-d7c710a812


Thank you @ralphbean and @eriol for working with us on this. :)


:heart: Indeed, thanks so much (and thanks @warsaw and @dstufft as well!).


On fedora, now it's showing this error on update to this package
Running transaction
  Upgrading   : python-requests-2.7.0-7.fc22.noarch                         1/2 
Error unpacking rpm package python-requests-2.7.0-7.fc22.noarch
Error unpacking rpm package python-requests-2.7.0-7.fc22.noarch
error: unpacking of archive failed on file /usr/lib/python2.7/site-packages/requests/packages/chardet: cpio: rename
python-requests-2.7.0-7.fc22.noarch was supposed to be installed but is not!


@ralphbean Theories?


People have been commenting on this bug about this error in fedora
https://bugzilla.redhat.com/show_bug.cgi?id=1236312


@vvro At this point there is no evidence that requests did anything to introduce this bug: it was almost certainly introduced downstream. It is _possible_ that this issue caused it, but that remains a Fedora problem. I highly recommend centralising the discussion of this problem on the Red Hat bugzilla, rather than here.


Agreed.  'cpio: rename' issues are almost always a result of having done a 'sudo pip install requests' on the system and introducing ownership conflicts with the rpm management system -- dueling package management systems.


Sorry for the horrible delay, I had to change my usual workflow after the git migration, but a positive note is that now, after the usual testing, I'm also rebuilding some dependants (I had to speedup my setup to rebuild all of them) of requests.

Debian patch: https://anonscm.debian.org/cgit/python-modules/packages/requests.git/commit/?id=6c882aca203bd33c129e2b4e1e4f82886e4b1fe0

The package will uploaded soon: unfortunately although now I'm officially a Debian Maintainer my sponsor had problems to give me upload permissions. He tried but he got an error, and we will investigate during these days.


Uploaded: https://packages.qa.debian.org/r/requests/news/20151028T232257Z.html.


Alright, that's Debian and Fedora handled. Thanks so much @ralphbean @warsaw @eriol, we :heart: you greatly.


Thanks @eriol and I'm always happy to help sponsor.  One thing we need to keep in mind is that as we update requests' dependencies, we have to make sure to also update requests setup.py.  I don't think we have any mechanisms in Debian to ensure this but we can probably rely on Ubuntu's -proposed migration to keep track of mismatches: http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html

The other thing we may want to think about is DEP-8 tests to make these verifications.  Then at least we'll see problems on qa.debian.org.  A README.Debian is a good idea too, but they are easy to miss :(


@warsaw it's a pleasure work with you! Also reading your hints! I had the same thought on README.Debian, but I will add it in the next upload.

Ubuntu's -proposed migration is very interesting! I will keep an eye on it for my packages.

After the git migration I updated my workflow and I discovered some useful tools.
Before the last requests' upload I tried to use [ratt](https://github.com/Debian/ratt) on my laptop to rebuild all the reverse dependencies: it was a bit troublesome since one of the rdeps is blender and it take more than 1 hour to build, so I did not rebuild all of them but only some.
To "fix" my building problems I was suggested to use debomatic: https://debomatic.github.io/ :smile: I'm reading the documentation right now.
The only problem is that it's not automated, but I will start to use it for all my packages before uploading them on the archive.

What about a more draconian approach in addiction to DEP-8 test? I'm thinking about testing at build time and let the package fail to build if the setup.py was not correctly updated.

Do you have any hints about safely parse a setup.py? Thanks!