Repository: pvlib/pvlib-python
Pull request #1459 (2022-05-19 09:25:14)

Allow read_surfrad to parse http files
<!-- Thank you for your contribution! The following items must be addressed before the code can be merged. Please don't hesitate to ask for help if you're unsure of how to accomplish any of the items. Feel free to remove checklist items that are not relevant to your change. -->

 - [x] I am familiar with the [contributing guidelines](https://pvlib-python.readthedocs.io/en/latest/contributing.html)
 - [x] Tests added
 - ~~[ ] Updates entries in [`docs/sphinx/source/reference`](https://github.com/pvlib/pvlib-python/blob/master/docs/sphinx/source/reference) for API changes.~~
 - [x] Adds description and name entries in the appropriate "what's new" file in [`docs/sphinx/source/whatsnew`](https://github.com/pvlib/pvlib-python/tree/master/docs/sphinx/source/whatsnew) for all changes. Includes link to the GitHub Issue with `` :issue:`num` `` or this Pull Request with `` :pull:`num` ``. Includes contributor name and/or GitHub username (link with `` :ghuser:`user` ``).
 - [x] New code is fully documented. Includes [numpydoc](https://numpydoc.readthedocs.io/en/latest/format.html) compliant docstrings, examples, and comments where necessary.
 - [x] Pull request is nearly complete and ready for detailed review.
 - [x] Maintainer: Appropriate GitHub Labels and Milestone are assigned to the Pull Request and linked Issue.

<!-- Brief description of the problem and proposed solution (if not already fully described in the issue linked to above): -->
A [StackOverflow user pointed out](https://stackoverflow.com/q/72283334/8558146) that the SUFRAD files are not only accessible from the FTP server but also via [HTTPS links](https://gml.noaa.gov/aftp/data/radiation/surfrad/Boulder_CO/2020/). This PR updates the ``read_surfrad`` function to allow for parsing remote HTTP files.


AdamRJensen: @kanderso-nrel Is a whatsnew entry needed?
kanderso-nrel: > @kanderso-nrel Is a whatsnew entry needed?

I can imagine a user being interested in this change, so I'd vote yes to a whatsnew entry :)
AdamRJensen: Here's the response I got from Chris Cornwall from NOAA in regards to the reason for the SURFRAD files being available on both the FTP server and the new HTTP index:

> HTTP access to our FTP data was set up a couple years ago because more and more web browsers ended support for the FTP protocol, in the interest of security. As you noticed, the same data are available on our [aftp.cmdl.noaa.gov](http://aftp.cmdl.noaa.gov/) server as through the web link https://gml.noaa.gov/aftp/data/radiation/surfrad/
> The reason for duplicate directories for the SURFRAD permanent locations is for backward compatibility to the old FTP data directory structure. The 3-letter site abbreviations are actually symbolic links on the server, pointing to their longer-named counterparts: dra -> Desert_Rock_NV, for example.
AdamRJensen: > any reason this is still sitting here?

I think it was just needing a second review. Feel free to merge.

Patch:
diff --git a/docs/sphinx/source/whatsnew/v0.9.2.rst b/docs/sphinx/source/whatsnew/v0.9.2.rst
index 028b905..812aee9 100644
--- a/docs/sphinx/source/whatsnew/v0.9.2.rst
+++ b/docs/sphinx/source/whatsnew/v0.9.2.rst
@@ -8,6 +8,9 @@ Deprecations
 
 Enhancements
 ~~~~~~~~~~~~
+* :py:func:`pvlib.iotools.read_surfrad` now also accepts remote files
+  with https links in addition to files on the SURFRAD FTP server
+  (:pull:`1459`)
 * Add :py:func:`pvlib.tracking.calc_surface_orientation` for calculating
   single-axis tracker ``surface_tilt`` and ``surface_azimuth`` from
   rotation angles. (:issue:`1471`, :pull:`1480`)
@@ -47,8 +50,9 @@ Requirements
 
 Contributors
 ~~~~~~~~~~~~
+* Adam R. Jensen (:ghuser:`AdamRJensen`)
 * Naman Priyadarshi (:ghuser:`Naman-Priyadarshi`)
 * Chencheng Luo (:ghuser:`roger-lcc`)
 * Prajwal Borkar (:ghuser:`PrajwalBorkar`) 
 * Kevin Anderson (:ghuser:`kanderso-nrel`)
-* Cliff Hansen (:ghuser:`cwhanse`)
\ No newline at end of file
+* Cliff Hansen (:ghuser:`cwhanse`)
diff --git a/pvlib/iotools/surfrad.py b/pvlib/iotools/surfrad.py
index b819923..7f4d86e 100644
--- a/pvlib/iotools/surfrad.py
+++ b/pvlib/iotools/surfrad.py
@@ -44,7 +44,7 @@ def read_surfrad(filename, map_variables=True):
     Parameters
     ----------
     filename: str
-        Filepath or url.
+        Filepath or URL. URL can be either FTP or HTTP.
     map_variables: bool
         When true, renames columns of the Dataframe to pvlib variable names
         where applicable. See variable :const:`VARIABLE_MAP`.
@@ -113,7 +113,8 @@ def read_surfrad(filename, map_variables=True):
     =======================  ======  ==========================================
 
     See README files located in the station directories in the SURFRAD
-    data archives [2]_ for details on SURFRAD daily data files.
+    data archives [2]_ for details on SURFRAD daily data files. In addition to
+    the FTP server, the SURFRAD files are also available via HTTP access [3]_.
 
     References
     ----------
@@ -122,8 +123,10 @@ def read_surfrad(filename, map_variables=True):
        `SURFRAD Homepage <https://www.esrl.noaa.gov/gmd/grad/surfrad/>`_
     .. [2] NOAA SURFRAD Data Archive
        `SURFRAD Archive <ftp://aftp.cmdl.noaa.gov/data/radiation/surfrad/>`_
+    .. [3] `NOAA SURFRAD HTTP Index
+       <https://gml.noaa.gov/aftp/data/radiation/surfrad/>`_
     """
-    if str(filename).startswith('ftp'):
+    if str(filename).startswith('ftp') or str(filename).startswith('http'):
         req = Request(filename)
         response = urlopen(req)
         file_buffer = io.StringIO(response.read().decode(errors='ignore'))


Test patch:
diff --git a/pvlib/tests/iotools/test_surfrad.py b/pvlib/tests/iotools/test_surfrad.py
index 6ef9fca..83f7ec7 100644
--- a/pvlib/tests/iotools/test_surfrad.py
+++ b/pvlib/tests/iotools/test_surfrad.py
@@ -7,6 +7,8 @@ from ..conftest import DATA_DIR, RERUNS, RERUNS_DELAY
 testfile = DATA_DIR / 'surfrad-slv16001.dat'
 network_testfile = ('ftp://aftp.cmdl.noaa.gov/data/radiation/surfrad/'
                     'Alamosa_CO/2016/slv16001.dat')
+https_testfile = ('https://gml.noaa.gov/aftp/data/radiation/surfrad/'
+                  'Alamosa_CO/2016/slv16001.dat')
 
 
 @pytest.mark.remote_data
@@ -19,6 +21,17 @@ def test_read_surfrad_network():
     assert local_data.equals(network_data)
 
 
+@pytest.mark.remote_data
+@pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
+def test_read_surfrad_https():
+    # Test reading of https files.
+    # If this test begins failing, SURFRAD's data structure or data
+    # archive may have changed.
+    local_data, _ = surfrad.read_surfrad(testfile)
+    network_data, _ = surfrad.read_surfrad(https_testfile)
+    assert local_data.equals(network_data)
+
+
 def test_read_surfrad_columns_no_map():
     data, _ = surfrad.read_surfrad(testfile, map_variables=False)
     assert 'zen' in data.columns