Third party libraries committed verbatim

Hi, I'm quite new to the world of Python, so might be missing something. When I submitted [this pull request](https://github.com/kennethreitz/requests/pull/1382) the other day, @sigmavirus24's response was surprising. Surely, putting a checkout of a third party library in your repository is not the right way to manage dependencies?

Can't you use [requirements.txt](https://github.com/kennethreitz/requests/blob/master/requirements.txt) to include urllib3 or at least make it a git submodule?


Hi @tasuk, thanks for providing that pull request and for opening this issue!

You are right, there are a few good ways to manage dependencies. What Requests does is to 'vendor' its dependencies: that is, to provide them as part of the library itself. The main reason we do this is because it has the effect of pinning us to a single version of our dependencies, without limiting what other users or libraries install.

Let me provide an example. If we moved urllib3 to `requirements.txt`, we would need to pin a specific version (to avoid being broken when/if urllib3 does a major version update that changes the API). So we'd add something like `urllib==1.5`. This all works great until you install another library that also uses urllib3, and also pins a version, e.g. `urllib==1.6`. At this stage, one of your libraries is going to be using a version of urllib3 that it wasn't expecting, and runs the risk of breaking. 

You see that Requests breaks and, being the good user that you are, you open an issue saying 'Requests broke over upgrade'. We run around like headless chickens for a while, and then say 'uh, none of us can reproduce this'. Eventually, after some amount of wasted time we discover that you're using a version of urllib3 that we don't support. Then we say "there's nothing we can do about this, you'll have to change all the source code of Requests yourself or wait until we update the version of urllib3 we test against".

As you can see, this is _crap_. So we can't be doing that.

As for git submodules, yes, we could in principle use that. Unfortunately, the general consensus is that git submodules are [a bit crap](http://codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/). Even if they were good, we'd gain very little from that: once again, we're pinned to a specific version of urllib3, so the ability to bring in its updates is not helpful. It's easier just to grab tagged versions of urllib3 and call it a day. =)

Does that answer your question?


To follow on, we had a case of exactly this problem at the start of the week. Python 3.3.2 changed the cookies API from Python 3.3.1, and broke Requests: we got two issues raised almost immediately and had to rush out a point release to fix it. Upgrading something that seems to be unrelated to Requests shouldn't break it. =)


Git submodules aren't even just a bad idea for the reason you posted @Lukasa, they include the entire repository not just the library we're looking for. In other words, let's say we vendored urllib3 as a submodule, the structure would be as follows

```
requests/
 | + .git/
 | - setup.py
 | + requests/
    | + packages/
       | - __init__.py
       | + urllib3/
          | + .git/
          | - setup.py
          | + urllib3/
       | + charade
          | + .git/
          | - setup.py
          | + charade/
```

So then we wouldn't be able to do: `from .packages import urllib3` or `from .packages import charade`. We couldn't even do `from .packages.urllib3 import urllib3` because the repository itself is not a package.


Oh man, I hadn't even noticed that. Yeah, that's really the kicker, isn't it: submodules aren't just crap, they aren't a solution to this problem at all. =D


This really isn't up for discussion.


@kennethreitz: Of course not. =) I was explaining why we do it, rather than suggesting that we'd ever change. Given @tasuk is relatively new to this, though, it seemed like explaining why lots of libraries do this.


Yeah, you're doing an amazing job :)


@Lukasa it could be fixed by structuring the project like logilab structures theirs (example: [astng](https://bitbucket.org/logilab/astng/src)) but I really really hate that structure.

And @kennethreitz, we're just trying to teach our new friend. (At least I hope he'll be our friend.)


Maybe if we give him a :cookie: 


We have cookies?! :open_mouth: Why wasn't I told about this? 


:tada:


@Lukasa, thank you for a very comprehensive answer!

So, basically, dependency management is also broken in Python :)

Coming from an inferior language where we're facing similar problems, I was hoping someone solved dependencies in Python. Maybe it's one of those problems that can never really be solved. I guess you'd have to be able to have several versions of one package installed in parallel and choose which one you're importing. Polluting imports with version numbers doesn't seem like the right thing to do though.

As for submodules, I know they're problematic (been there seen that), but I don't see why the package root shouldn't be the project root (though Python people don't seem to do that). That looks like lesser evil than including whole 3rd party libs in your repo. I'm a bit surprised that's standard practice in Python-world.

It's an interesting situation, each of the solutions sucks in its own specific way. In PHP world, we use Composer (our version of "pip -r"). Sometimes we get impossible-to-resolve conflicts and sometimes dependency updates break stuff (though there should be no api changes for minor releases), but we learned to live with it.

One more question &ndash; what is the reason to use requirements.txt for `pytest` and `invoke`, but not for `urllib3`?

@sigmavirus24 yes I'll be your github-friend, @kennethreitz thanks for the cookie!


Actually, the right solution is to have requirements.txt install the dependencies locally in a recursive way, so that each of the packages using requirements.txt can have their own specific version of each dependency.


Actually @tasuk it isn't necessarily common practice to vendor dependencies. @kennethreitz uses it here because of how incredibly popular requests is and what a headache it would be to have different versions of dependencies floating around.

On the other hand, most of my libraries (the ones that have dependencies) use pip to install the dependencies since (for the most part) the libraries I use I actively help develop, so I can anticipate changes that my cause issues.

As for requirements.txt, we use that for [Travis CI](travis-ci.org) to install dependencies for testing requests. Those aren't installation dependencies at all. Since that's only used for testing, and we only tests requests, we have no need to use our vendored libraries' requirements.txt files simply because we aren't running their test suites.

Some people us requirements.txt to also determine what needs to be installed for their package in general, but that isn't it's only use, just like not all projects vendor dependencies. :)


@sigmavirus24 ah cool, that explains everything!


@sigmavirus24 has covered most of the points I was going to. =)

Like him, my projects overwhelmingly use Pip to manage dependencies, for the exact same reason. I have written a few requests plugins, and because I am a requests contributor I can predict problems that might come up. Though even then, I pin to a minimum Requests version, so I'm not totally immune. =P

You raised making the package root be the project root. There is a very good reason we don't do that in Python, and that reason is because in Python, folders define your namespaces. If you look at the Requests project root, you'll see it looks like this:

```
requests/
|- docs/
|- ext/
|- requests/ (the package root)
|- .gitignore
|- .travis.yml
|- AUTHORS.rst
|- HISTORY.rst
|- LICENSE
|- MANIFEST.in
|- NOTICE
|- README.rst
|- invokefile.py
|- requirements.txt
|- setup.py
|- tasks.py
|- test_requests.py
```

If we put an `__init__.py` here, turning this into the package root, the following lines of code would become possible:

``` python
from requests import setup
from requests import tasks
from requests import test_requests
from requests import docs
from requests import invokefile
from requests import ext
```

We don't want people to be able to import any of these things: they aren't part of the library itself. But we need to be able to keep them under version control as well, because they are still very important. For this reason, we use a subfolder as the root of our package, which allows us to easily manage the other package metadata.

(A note: the fact that we have our testing dependencies in `requirements.txt` annoys me: I don't do it in my own projects. However, it's really minor so I just let it go. =) )


In fact, I'd go further: the standard python distribution tools (Distutils and Setuptools) essentially _mandate_ that you don't ship the root of your project as the root of your package. You need that `setup.py` script, and it can't be in your package.


On Fri, May 24, 2013 at 06:28:18AM -0700, Cory Benfield wrote:

> We don't want people to be able to import any of these things: they aren't 
> part of the library itself. But we need to be able to keep them under 
> version control as well, because they are still very important. For this 
> reason, we use a subfolder as the root of our package, which allows us to 
> easily manage the other package metadata.

Also note that Ruby does the same thing, a typical ruby structure is:

```
requests.rb/
| - lib/
  | - requests/
  | - requests.rb
| - spec/
| - Gemfile
| - README.md
| - requests.gemspec
```

So it isn't just a python thing :)

Also, you could look at C projects which typically break stuff up into:

```
project/
| - src/
  | - library_1/
    | - Makefile
  | - library_2/
    | - Makefile
| - lib/
| - Makefile
| - configure
| - README.txt
| - LICENSE.txt
```

But this is far less "standardized" than Ruby or Python. Also note that 
dependency management in Ruby is just as bad as it is in python (I think). And 
dependency management in Ubuntu (Debian, Arch, etc.) suffers the same issues.

> (A note: the fact that we have our testing dependencies in 
> `requirements.txt` annoys me: I don't do it in my own projects. However, 
> it's really minor so I just let it go. =) )

Now I have to look more closely at your projects to see what you do. I use it 
for github3.py but should stop since I've started packaging the tests with the 
tarball and put the test dependencies in the `setup.py` file. Anyway, this 
isn't useful for this issue anyway, especially since it's closed. :)


@Lukasa, that all makes perfect sense.

Even in PHP-land it's usual to separate things in a similar way (see e.g. [guzzle](https://github.com/guzzle/guzzle)). But we have funky class-loading mechanisms instead of adhering to the directory structure for namespacing, allowing us to be a bit more lenient about which files go where.

To reiterate my previous point - wouldn't it be wonderful if pip was able to manage packages locally in a recursive way? ("I need tag 3.1 of package 'x' and commit 45fe8a2 of package 'y', can you put them in requests/packages for me?")


@Lukasa 

> A note: the fact that we have our testing dependencies in `requirements.txt` annoys me (...)

I'm glad it annoys you as it annoys me, too :) If these are testing only dependencies I think the file should be named `requirements-testing.txt` or something similar to make it clear what kind of dependencies it enumerates.