An Efficient Way to Read Data from the Web Directly into Python | by Luke Gloege, Ph.D. | Towards Data Science

Storage space on my hard drive is precious and I don’t want to download a bunch of data when I’m just going to process it down into something manageable.
Sometimes we can paste a URL into pd.read_csv() or xr.read_dataset() and it will gladly read the data. For example, xarray includes support for OPeNDAP to access some, but not all, datasets over HTTP.
This post will describe one solution you can use when things aren’t copacetic and directly pasting the URL into xarray fails. I will outline how to read data from a web server directly into Python, even from a zip file, all without writing anything to disk.
The goal is to access data across HTTP or FTP, which uses a request and response structure. For example, we could request the contents of a URL and the server will (hopefully) send us back the data as a response. Here is how this structure works in the urllib.request Python package:
url='https://google.com': this is the URL we want to access.req=urllib.request.Request(url): creates a Request object specifying the URL we want.resp=urllib.request.urlopen(resp): returns a response object from the server for the requested URL.data=resp.read(): the response object (resp) is file-like, which means we can read it.
url='https://google.com': this is the URL we want to access.
req=urllib.request.Request(url): creates a Request object specifying the URL we want.
resp=urllib.request.urlopen(resp): returns a response object from the server for the requested URL.
data=resp.read(): the response object (resp) is file-like, which means we can read it.
Now let’s apply this basic idea to some real data.
The code below accesses a specific year of air temperature from the NCEP reanalysis. I always prefer to use a context manager when opening a URL so I don’t forget to close it.
The last line on this code returns a xarray dataset with our data. However, that line is a little busy, let’s unpack it:
resp.read(): is our requested data. However, it is in bytes.
io.BytesIO(): Keeps our requested data as bytes in an in-memory buffer.
xr.open_dataset(): opens up the bytes file as a xarray dataset. I can do this because the URL I requested is a NetCDF file.
This is great! With minimal effort, we were able to read data into memory without downloading it.
If you get an error saying “can’t open NetCDF as bytes,” then you need to install h5netcdf.
conda install -c conda-forge h5netcdf
Not a problem, the Python zipfile package can take care of that. In the code below I am requesting a zip file from a server. I then use the zipfile package to list the files inside. Specifically, I use the namelist() method. Finally, I use xarray to read the contents of one of the NetCDF files contained inside into a dataset.
If you ever request access and get this error message it means the server knows you are trying to access it from a program. However, we are smarter than a computer and can trick it into thinking we are a web browser instead. This can be done by attaching a User-Agent to our request so it looks like it is coming from a web browser. As an analogy, this is like knocking on a door and announcing who we are. Attaching a different User-Agent is analogous to impersonating somebody we’re not.
When you find a dataset on the web there will usually be a button to download the dataset. We want to know the URL linking to that data. In most browsers, you can right-click and on the download link and “Copy Link Address.” Here is an example using Brave browser to get the link to an oceanographic dataset.
This approach can streamline data pipelines and make your code easily reproducible by others. However, there are some drawbacks. For instance, you will not be able to access the data if the server is down. This approach may not work in all situations, such as if the dataset is too big to fit into memory. I haven’t run into any issues yet, but I’m not sure how well this approach would work for large datasets that can still fit into memory.
All in all, I find this to be a clean way to make my code reproducible by other researchers. The benefit is I am not relying on them to first download the data. At the very least this is another trick to add to your toolbox.
I am happy to help troubleshoot any issues you may have reading data into Python from a server.
Thank you for reading and supporting Medium writers