Support 'file' scheme

I was using urllib on a project and decided to change to requests, but I was using a mix of 'http' and 'file' schemes on some scenarios which is handled by urllib. I know requests is mostly about network and doesn't make much sense to use for files but transparency would be useful.

Would this be possible?


I'm pretty sure this support would have to be added to urllib3 (or if it's already there, the vendorized package would have to be updated). @shazow might be a better person to talk to about this. In the meantime, I'll see if it's part of urllib3.


It could be done on the requests side since `file` scheme won't be doing any network-based work nor does it require any connection pooling. For those reasons I'm also uncomfortable adding it into urllib3. :/ What do you think, @kennethreitz?


-100. Once we have transport adapters, this will be really easy to write, but this is currently one of the biggest problems with urllib2. Massive security exploit.


I use requests to do scraping and we have html files of the pages we scrape that we want to use for tests. We wanted to just tell requests the path to these files instead of the URL of the site, but because of this bug we need to go write some custom code now to check the scheme, using requests accordingly.

This means we have to add functionality to our codebase to support our tests, which is certainly not ideal. 

Don't know what transport adapters are, so can't comment on that, but thought I'd put a use case in here in case it's useful. 


You have two options. The first is to write a custom Transport Adapter, which isn't very hard. Unfortunately, this won't work in the current version of Requests, which insists you have both a netloc (e.g. hostname) and a path. A `file` URL does not have a netloc, so we throw exceptions.

The second option is to run a local webserver to host the files for testing. You can do that very easily: go to the directory the files are in and run `python -m SimpleHTTPServer`.


I tried using the SimpleHTTPServer as mentioned above, but I don't want to have to start it up before running my tests, and if you start it as part of the setUp or setUpClass method in your unittests, there's no way to tear it down in the tearDown method. This leaves you with frustrating issues around tied up addresses and "Address already in use" exceptions. Beyond that, starting up a server just to do local file access seems like the wrong approach in a unittest. 

Not sure how helpful this is, but I wrote a MockRequest object that can be used when the method is set to LOCAL rather than get or post:

```
from requests.models import Request, Response
from requests.exceptions import ConnectionError


class MockRequest(Request):
    def __init__(self, url=None):
        super(Request, self).__init__()
        self.url = url

    def get(self):
        r = Response()
        try:
            r._content = open(self.url).read()
            #: Integer Code of responded HTTP Status.
            r.status_code = 200
        except IOError as e:
            r.status_code = 404
            raise ConnectionError(e)

        r._content_consumed = True

        #: Final URL location of Response.
        r.url = self.url

        #: Resulting :class:`HTTPError` of request, if one occurred.
        self.error = None

        #: Encoding to decode with when accessing r.content.
        self.encoding = None

        #: The :class:`Request <Request>` that created the Response.
        self.request = self

        # Return the response.
        return r
```

This is currently working in my unit tests, where I can easily change the request.method to "LOCAL", and then use the MockRequest object instead of a standard one. I only took the most cursory dive into the requests library, but maybe something like this would provide a useful tool to people using requests in testing environments. 

Is this something that could be added to Requests, in lieu of the other options above, which it seems wont' be moving forward any time soon? 


I'm glad you got something working for your use case, @mlissner!

I doubt Requests will ever have direct support in the main library for the `file:` URL scheme. Aside from Kenneth's point above about it being a security risk, Requests is strictly an HTTP library. `file:` URLs do not use HTTP, so they're outside the Requests use case.

If you wanted a more transparent example, you should be able to use something like this:

``` python
import requests
from cStringIO import StringIO

class TestAdapter(requests.adapters.HTTPAdapter):
    urlmap = {'http://www.google.com/', '/home/lukasa/test.html'}

    def send(self, request, **kwargs):
        try:
            rsp = open(urlmap[request.url])
            status = 200
        except IOError:
            rsp = StringIO()
            status = 404

        response = requests.Response()
        response.status_code = status
        response.headers = {}
        response.encoding = None
        response.raw = rsp
        response.url = request.url
        response.request = request
        response.connection = self

        return response
```

Mounting that transport adapter _should_ (I haven't tested it) provide you with the behaviour you want.


Thanks @Lukasa. I gave that a whirl and tried to make it work for a bit, but I ran into an error I can't get past.

This line is the one causing trouble: https://github.com/kennethreitz/requests/blob/master/requests/cookies.py#L109

From what I can tell, setting response.raw to a file or StringIO object won't work because: 

``` python
Traceback (most recent call last):
  File "/home/mlissner/Programming/intellij/juriscraper/tests/tests.py", line 43, in test_scrape_all_example_files
    site.parse()
  File "/usr/lib/python2.7/dist-packages/juriscraper/GenericSite.py", line 85, in parse
    self.html = self._download()
  File "/usr/lib/python2.7/dist-packages/juriscraper/GenericSite.py", line 238, in _download
    r = s.get(self.url, headers={'User-Agent': 'Juriscraper'})
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 347, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 446, in send
    extract_cookies_to_jar(self.cookies, request, r.raw)
  File "/usr/local/lib/python2.7/dist-packages/requests/cookies.py", line 108, in extract_cookies_to_jar
    res = MockResponse(response._original_response.msg)
AttributeError: 'file' object has no attribute '_original_response'
```

I'm not sure how to proceed with this, though I read your FTP Adapter code and blog entry (but there's not much else on transport adapters yet!). I'm tempted to say that not allowing file-like objects is a bug, but I know that's been closed before as a wont-fix. I've also gotten the impression that _original_response shouldn't be used for anything (though it is used in cookie.py). So...I'm confused how to make this work properly, whether _original_response shouldn't be used the way it is, and what the proper fix for this should be.


That's an interesting bug. Could you perhaps trying using @kanzure's 
requestions? I suspect that's probably getting a little long in the tooth but 
it probably works around that issue if I remember it correctly. I forget, 
however, whether it uses transport adapters.


@sigmavirus24 I took a look at requestions, but don't see any use of adapters (or workarounds) in there. Regardless, I'd rather avoid the dependency if possible.


Roger that.


I can confirm that I don't remember if I added support for adapters in kanzure/requestions.