Importing requests is fairly slow

We are using requests in our projects, and it is working great. Unfortunately, for our CLI tools, using requests is an issue because it is slow to import. E.g. on my 2014 macbook:

```
python -c "import requests"
```

Takes close to 90 ms.

Is optimizing import time worth of consideration for the project ?


This is almost certainly the result of CFFI. Can you print what libraries you have installed in your environment (`pip freeze`)?


This is not linked to cffi (I don't have it installed). I did the test w/ a quasi empty virtualenv:

```
$ pip list
pip (8.1.2)
requests (2.10.0)
setuptools (21.1.0)
wheel (0.24.0)
$ time python -c "import requests"

real    0m0.090s
user    0m0.061s
sys 0m0.026s
```


I have a 2012 Macbook pro:, using a slightly different way of measuring the import time:

```
$ pip list
pip (8.0.2)
requests (2.10.0)
setuptools (19.6.2)
wheel (0.26.0)
$ python -m timeit -n 10000000 'import requests'
10000000 loops, best of 3: 0.571 usec per loop
```

I also upgraded pip and setuptools to match your environment:

```
$ pip list
pip (8.1.2)
requests (2.10.0)
setuptools (21.1.0)
wheel (0.26.0)
$ python -m timeit -n 10000000 'import requests'
10000000 loops, best of 3: 0.59 usec per loop
```

Further

```
$ time python -c 'import requests'
python -c 'import requests'  0.07s user 0.05s system 61% cpu 0.200 total
```

Which is still not ideal, but is probably higher than the simple import of requests due to Python's initialization.


That said, if you have information about specific things in our `__init__.py` (and elsewhere) that are causing slowness, I think we can fix them so long as they don't break backwards compatibility or our functional API.


`timeit` is not informative for imports, you have to create a new process every time because of python import caching.

The main cost is importing urllib3, so I suspect that should be addressed there.


@sigmavirus24 as @cournape mentions you can't time it like that; try using `time` on the command line. On my machine (ran each ~6-10 times to try to get a reliable average) for user+sys times (including sys because if there's any strange calls made into the kernel as a result of the import, that should be counted):
- `time python -c ""` (CPy 2.7.11) = 130 ms (time for Python to start up)
- `time python -c "import requests"` (2.10.0) = 240 ms (above + importing requests)
- `time python -c "import urllib3"` (1.15.1) = 210 ms (installed separately)

So `urllib3` takes about 80 ms, then about 30 ms more for requests stuff.


Neither of these numbers seem that high to me. 


Yeah. Especially as your average request is going to take a few hundred milliseconds or so. If you _reeeeally_ want to shave off milliseconds, could always back-translate to use the stdlib. 😜 

Or you could shorten the name of your command line tool by 1-2 characters.


Ah, I didn't realize you were working on a CLI tool. Well, it hasn't seemed to be an issue for httpie.


It is true that as soon as you do even one request on a remote server, the delay will not matter much. But the problem in python is that you pay the import cost whether you actually do a request or not: think doing `tool --version`, `tool --help`, etc... Imagine a tool like git or hg that only occasionally do http IO.

80 ms is pretty bad in this case: the rule of thumb for a CLI tool to have noticeable delay is ~100-150 ms. Try using e.g. git with a 200 ms delay, it is a bit annoying.

Now, I completely understand that it may not be a focus of the library, especially if it is not that easy to improve.


I personally think you're using the wrong tool for the job — given that, on the metrics given above, Python itself is taking 120ms alone. Httpie and mercuial are well-loved tools that do their jobs very well, but they are not, nor ever will they be, curl or git. 


Makes sense; you could code up some lazy loading bits. Found this though, which claims to have been excerpted from `hg`: https://github.com/bwesterb/py-demandimport (haven't ever used it, no guarantees). 

Apparent origin: https://selenic.com/hg/file/tip/mercurial/demandimport.py


@kennethreitz python does not take 120 ms to start, unless you are on a seriously broken environment. It is much closer to 20 ms on decently modern hw (< 5 years), i.e. importing requests means 3x the cost of starting python.

FWIW, on my 2011 Desktop PC (Debian):

```
$ time python -c ""
real    0m0.020s
user    0m0.016s
sys 0m0.000s
$ time python -c "import requests"
real    0m0.107s
user    0m0.088s
sys 0m0.012s
```

On my 2014 macbook (OS X)

```
$ time python -c ""

real    0m0.022s
user    0m0.010s
sys 0m0.009s
$ time python -c "import requests"
real    0m0.096s
user    0m0.060s
sys 0m0.032s
```

A simple `hg` (for help) on my macbook takes ~ 100 ms


Oh, oops, yeah...my environment is a bit wonky because of the `pyenv` shim (adds about 100 ms because it's a bash script that leads to python with `exec`). Bypassing that, the incremental times are still the same and Python itself loads in 10 ms or so.


haha, I thought that number looked high :)


The "profimp" module shows where the time goes:

https://pypi.python.org/pypi/profimp

pkg_resources takes up a lot of time in general.

Another time suck is the automatic loading of `.packages.urllib3.contrib.pyopenssl`. It would be great if we could opt-out, as there is core SNI support in the appropriate versions of Python.

There doesn't appear to be any way of doing this today, if PyOpenSSL is importable. Would a PR allowing an opt-out be likely to be merged?

@dsully How would you propose to expose the API for an opt-out?

@Lukasa Unless I'm mistaken, the `.contrib.pyopenssl` ssl wrapper & context is not needed on Python 2.7.9+ and 3.4+. Given that, `requests` is always using pyopenssl when it is installed, even when the core Python libs support SNI, etc. So, changing the import to be:

```
# Attempt to enable urllib3's SNI support, if possible & needed.
try:
    from ssl import SSLContext  # Python 2.7.9+, 3.4+
except ImportError:
    try:
         from .packages.urllib3.contrib import pyopenssl
         pyopenssl.inject_into_urllib3()
    except ImportError:
         pass
```

Less of an opt-out, more of a not-needed.

Now, if there are cases where the pyopenssl SSLContext wrapper is desired for some reason (?) even when the core libraries are sufficient, then I'll make another pass. What do you think?

@dsully you said:

> pkg_resources takes up a lot of time in general.

Where is Requests using pkg_resources? I don't believe either Requests or Urllib3 uses it.

> Now, if there are cases where the pyopenssl SSLContext wrapper is desired for some reason (?) even when the core libraries are sufficient

If I remember correctly, only the more recent versions of 2.7 are actually appropriate. I think there were some minor issues with 2.7.9 and 2.7.10.

> Unless I'm mistaken, the .contrib.pyopenssl ssl wrapper & context is not needed on Python 2.7.9+ and 3.4+.

Unfortunately, that's not true. The best example is on the system Python on macOS, which provides an SSLContext but which ships OpenSSL 0.9.8zh. This is an almost impossible to use version which is missing a number of vital features, including TLSv1.2 support. This is resolved by PyOpenSSL, which uses the OpenSSL 1.0.2 statically linked inside the cryptography module. This is *vital* to secure systems.

*More generally*, using PyOpenSSL allows for users to link in a version of OpenSSL that is not constrained by the OpenSSL that their base Python is linked against.

@sigmavirus24 Indirectly via `cryptography`:

```
src/cryptography/hazmat/backends/__init__.py
7:import pkg_resources
27:            for ep in pkg_resources.iter_entry_points(
```

Imported via:

```
packages/urllib3/contrib/pyopenssl.py
49:from cryptography.hazmat.backends.openssl import backend as openssl_backend
50:from cryptography.hazmat.backends.openssl.x509 import _Certificate
```

@Lukasa Right.. forgot about that. I don't have that particular issue on macOS, but most people do.

I'll look at coming up with a way to explicitly opt-out then.

It should be noted that the *easiest* way to opt-out is to simply not have PyOpenSSL in your environment. If it's present and you need it, you're presumably paying that import cost somewhere anyway.

Also worth noting that `pkg_resources` takes longer to iterate over entry-points on systems with very large numbers of packages installed. It's also sadly the best solution to finding plugins defined by packages.

@Lukasa - yes and no. Our build environment has real dependency management for modules (https://engineering.linkedin.com/blog/2016/08/introducing--py-gradle--an-open-source-python-plugin-for-gradle), which means just because someone included PyOpenSSL in their dependency tree, doesn't mean that the code you are importing for your upstream uses it. Lots of code gets installed transitively, but not imported.

@reaperhulk BTW, is slow import of cryptography still an ongoing issue?

@Lukasa - without moving the location of the pyopenssl loader, currently in `requests/__init__.py`, would an environment variable be acceptable?

I'm *tentatively* open to that, yes.

> Lots of code gets installed transitively, but not imported.

That sounds like the real root cause of this. Well that and the possibility that cryptography is still slow to import. Which is an issue for pyca/cryptography.

> would an environment variable be acceptable?

I don't like that. Let's say someone is using HTTPie and notice it's slow. They're also using requests in a development project. If they export the environment variable to speed it up without understanding the ramifications *and* they're on an LTS distribution with a *terrible* `ssl` module and *terrible* OpenSSL, then they'll start seeing SSLErrors in their project if they were relying on any of the functionality provided by PyOpenSSL. But those SSLErrors weren't there yesterday and they've already forgotten that they exported that variable. Now we have yet one more avenue of complexity to help them debug. I abhor that.

@sigmavirus24 I hear you there. If environment changes aren't ok, an explicit call? I'd have to move the current injection, since it happens in `__init__.py`