Allow read_csv to take URLs
State: closed

This allows read_csv to take URLs. The tests are going to need to be modified after it's merged to point the the new repo URL or if you want to host a test file somewhere else. I also have no idea what the file:// path should be on non-posix systems, so this path might need some adjustment. Not sure. There's no test case for ftp, but I don't see why it wouldn't work the same as http.


Labels:

Comments:
On Python 3, the HTTP response produces bytes, but the parser implementation expects strings (i.e. unicode). I think the correct way would be to wrap the response in an [io.TextIOWrapper](http://docs.python.org/library/io#io.TextIOWrapper), using the encoding passed to `read_csv`.

what about numpy.compat.asstr or similar? io doesn't exist in python 2.5

`codecs.StreamReader` should exist in 2.5, but I think whatever we use needs an `if PY3` anyway, because the machinery for handling CSV files on Python 2 expects a bytes-mode file-like object.

`numpy.compat.asstr` looks like it's hardcoded to Latin-1, so I don't think that's quite right.

Is this what you have in mind?

Yes, that looks sensible, although I haven't tested it yet. I'd also specify `errors='replace'` when creating the wrapper, so that it will tolerate an incorrectly guessed encoding.

Hmm. Who guesses the encoding? The user? Or is there some BOM checking somewhere?

If the user, it seems to me I'd want to know if I'm not working with the encoding I think I am. Either way, you're probably going to have to either fix your text or pass another encoding. Replace makes it such that you wouldn't discover this until later right?

The user has the option of doing so, but I guess most of the time they're going to leave it at the default, which I believe follows the encoding specified by locale (UTF-8 for Mac & most Linux, a particular code page for Windows).

In general, I'd agree with failing early and loudly, but with encoding, either it's a tiny detail, and it's a pain to have to keep guessing at encodings when you don't really care, or the text will be obviously gibberish if you get it wrong.

I think for opening files, we use a compromise - if the user specifies an encoding, we use `errors='strict'` so it fails if they're wrong, but if they leave it at the default, we use `errors='replace'`. We could do the same here.

Added the error handling.

Sorry, having tested it, it turns out that `TextIOWrapper` doesn't play nicely with an http response - if you try to read it line by line, it closes after the first read, and the next line fails with `ValueError: I/O operation on closed file.` I've worked out what needs doing, I'll make a mini-PR against this PR.

Thanks. Updated the PR.

Great, then this is alright as far as I'm concerned, so I'll ping @wesm and @adamklein to look at it.

@jseabold things have become a bit of a merging mess after PR #952. would you mind taking a crack at merging this work onto pydata/master?

Yeah, I'll have a look.

Rebased on master and force pushed. Should be okay now as long as it doesn't screw up @takluyver 

thanks dude. everything looks OK (haven't run tests on py3 yet but will soon)

might want to add some logic (at soem point) to skip the url test in some cases, but maybe no big deal. i pointed it at pydata/pandas now