Repository: pvlib/pvlib-python
Pull request #1374 (2022-01-08 12:53:36)

Add variable mapping of psm3
<!-- Thank you for your contribution! The following items must be addressed before the code can be merged. Please don't hesitate to ask for help if you're unsure of how to accomplish any of the items. Feel free to remove checklist items that are not relevant to your change. -->

 - ~~[ ] Closes #xxxx~~
 - [x] I am familiar with the [contributing guidelines](https://pvlib-python.readthedocs.io/en/latest/contributing.html)
 - [x] Tests added
 - ~~[ ] Updates entries to [`docs/sphinx/source/api.rst`](https://github.com/pvlib/pvlib-python/blob/master/docs/sphinx/source/api.rst) for API changes.~~
 - [x] Adds description and name entries in the appropriate "what's new" file in [`docs/sphinx/source/whatsnew`](https://github.com/pvlib/pvlib-python/tree/master/docs/sphinx/source/whatsnew) for all changes. Includes link to the GitHub Issue with `` :issue:`num` `` or this Pull Request with `` :pull:`num` ``. Includes contributor name and/or GitHub username (link with `` :ghuser:`user` ``).
 - [x] New code is fully documented. Includes [numpydoc](https://numpydoc.readthedocs.io/en/latest/format.html) compliant docstrings, examples, and comments where necessary.
 - [x] Pull request is nearly complete and ready for detailed review.
 - [x] Maintainer: Appropriate GitHub Labels and Milestone are assigned to the Pull Request and linked Issue.

<!-- Brief description of the problem and proposed solution (if not already fully described in the issue linked to above): -->
Add `map_variables` argument to `get_psm3` and `read_psm3` functions to be consistent with the general iotools function pattern.

At the moment `map_variables` defaults to None (False) and a deprecation warning has been added. The PR is similar to what was done for `get_pvgis_tmy` in #1268.

Additionally, I have implemented it such that pvlib names can be used when specifying the `attributes` arguments (requested columns), using the variable map in reverse.

AdamRJensen: It seems the `zenith` column returned by psm3 is "apparent_zenith", see comparison below:

![image](https://user-images.githubusercontent.com/39184289/148665735-19af3dc9-221d-4cfc-ade0-d1f1a8594472.png)

However, should the psm zenith column map then be mapped to 'apparent_zenith' or 'apparent_solar_zenith'?

The pvliv [Variables and Symbols List](https://pvlib-python.readthedocs.io/en/stable/variables_style_rules.html) has 'solar_zenith' as the standard name, which is also what zenith is mapped to in the [get_cams()](https://github.com/pvlib/pvlib-python/blob/master/pvlib/iotools/sodapro.py#L18-L29) function.
mikofski: @AdamRJensen  sorry for bombing this issue with irrelevant request, but would you be interested in showing your work on IO tools at SciPy conference? https://www.scipy2022.scipy.org/
AdamRJensen: This is just a test to see if the maintainer tags work

@pvlib/pvlib-maintainer
@pvlib/maintainer
@kanderso-nrel


Patch:
diff --git a/docs/sphinx/source/whatsnew/v0.9.1.rst b/docs/sphinx/source/whatsnew/v0.9.1.rst
index 363fca1..8aa1abb 100644
--- a/docs/sphinx/source/whatsnew/v0.9.1.rst
+++ b/docs/sphinx/source/whatsnew/v0.9.1.rst
@@ -19,6 +19,8 @@ Deprecations
 
 Enhancements
 ~~~~~~~~~~~~
+* Added ``map_variables`` option to :py:func:`pvlib.iotools.get_psm3` and
+  :py:func:`pvlib.iotools.read_psm3` (:pull:`1374`)
 * Added `pvlib.bifacial.infinite_sheds`, containing a model for irradiance
   on front and back surfaces of bifacial arrays. (:pull:`717`)
 * Added ``map_variables`` option to :func:`~pvlib.iotools.read_crn` (:pull:`1368`)
@@ -75,3 +77,4 @@ Contributors
 * Johann Loux (:ghuser:`JoLo90`)
 * Saurabh Aneja (:ghuser:`spaneja`)
 * Jack Kelly (:ghuser:`JackKelly`)
+* Somasree Majumder(:ghuser:`soma2000-lang`)
diff --git a/pvlib/iotools/psm3.py b/pvlib/iotools/psm3.py
index 10ff919..a8f9781 100644
--- a/pvlib/iotools/psm3.py
+++ b/pvlib/iotools/psm3.py
@@ -1,4 +1,3 @@
-
 """
 Get PSM3 TMY
 see https://developer.nrel.gov/docs/solar/nsrdb/psm3_data_download/
@@ -8,6 +7,8 @@ import io
 import requests
 import pandas as pd
 from json import JSONDecodeError
+import warnings
+from pvlib._deprecation import pvlibDeprecationWarning
 
 NSRDB_API_BASE = "https://developer.nrel.gov"
 PSM_URL = NSRDB_API_BASE + "/api/nsrdb/v2/solar/psm3-download.csv"
@@ -20,12 +21,31 @@ ATTRIBUTES = (
     'surface_pressure', 'wind_direction', 'wind_speed')
 PVLIB_PYTHON = 'pvlib python'
 
+# Dictionary mapping PSM3 names to pvlib names
+VARIABLE_MAP = {
+    'GHI': 'ghi',
+    'DHI': 'dhi',
+    'DNI': 'dni',
+    'Clearsky GHI': 'ghi_clear',
+    'Clearsky DHI': 'dhi_clear',
+    'Clearsky DNI': 'dni_clear',
+    'Solar Zenith Angle': 'solar_zenith',
+    'Temperature': 'temp_air',
+    'Relative Humidity': 'relative_humidity',
+    'Dew point': 'temp_dew',
+    'Pressure': 'pressure',
+    'Wind Direction': 'wind_direction',
+    'Wind Speed': 'wind_speed',
+    'Surface Albedo': 'albedo',
+    'Precipitable Water': 'precipitable_water',
+}
+
 
 def get_psm3(latitude, longitude, api_key, email, names='tmy', interval=60,
              attributes=ATTRIBUTES, leap_day=False, full_name=PVLIB_PYTHON,
-             affiliation=PVLIB_PYTHON, timeout=30):
+             affiliation=PVLIB_PYTHON, map_variables=None, timeout=30):
     """
-    Retrieve NSRDB PSM3 timeseries weather data from the PSM3 API.  The NSRDB
+    Retrieve NSRDB PSM3 timeseries weather data from the PSM3 API. The NSRDB
     is described in [1]_ and the PSM3 API is described in [2]_, [3]_, and [4]_.
 
     .. versionchanged:: 0.9.0
@@ -48,19 +68,23 @@ def get_psm3(latitude, longitude, api_key, email, names='tmy', interval=60,
         PSM3 API parameter specifing year or TMY variant to download, see notes
         below for options
     interval : int, {60, 5, 15, 30}
-        interval size in minutes, must be 5, 15, 30 or 60.  Only used for
+        interval size in minutes, must be 5, 15, 30 or 60. Only used for
         single-year requests (i.e., it is ignored for tmy/tgy/tdy requests).
     attributes : list of str, optional
         meteorological fields to fetch. If not specified, defaults to
         ``pvlib.iotools.psm3.ATTRIBUTES``. See references [2]_, [3]_, and [4]_
-        for lists of available fields.
+        for lists of available fields. Alternatively, pvlib names may also be
+        used (e.g. 'ghi' rather than 'GHI'); see :const:`VARIABLE_MAP`.
     leap_day : boolean, default False
-        include leap day in the results.  Only used for single-year requests
+        include leap day in the results. Only used for single-year requests
         (i.e., it is ignored for tmy/tgy/tdy requests).
     full_name : str, default 'pvlib python'
         optional
     affiliation : str, default 'pvlib python'
         optional
+    map_variables: boolean, optional
+        When true, renames columns of the Dataframe to pvlib variable names
+        where applicable. See variable :const:`VARIABLE_MAP`.
     timeout : int, default 30
         time in seconds to wait for server response before timeout
 
@@ -96,14 +120,15 @@ def get_psm3(latitude, longitude, api_key, email, names='tmy', interval=60,
     +===========+=============================================================+
     | Year      | 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, |
     |           | 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, |
-    |           | 2018, 2019                                                  |
+    |           | 2018, 2019, 2020                                            |
     +-----------+-------------------------------------------------------------+
     | TMY       | tmy, tmy-2016, tmy-2017, tdy-2017, tgy-2017,                |
     |           | tmy-2018, tdy-2018, tgy-2018, tmy-2019, tdy-2019, tgy-2019  |
+    |           | tmy-2020, tdy-2020, tgy-2020                                |
     +-----------+-------------------------------------------------------------+
 
     .. warning:: PSM3 is limited to data found in the NSRDB, please consult the
-        references below for locations with available data.  Additionally,
+        references below for locations with available data. Additionally,
         querying data with < 30-minute resolution uses a different API endpoint
         with fewer available fields (see [4]_).
 
@@ -133,6 +158,13 @@ def get_psm3(latitude, longitude, api_key, email, names='tmy', interval=60,
     # convert to string to accomodate integer years being passed in
     names = str(names)
 
+    # convert pvlib names in attributes to psm3 convention (reverse mapping)
+    # unlike psm3 columns, attributes are lower case and with underscores
+    amap = {value: key.lower().replace(' ', '_') for (key, value) in
+            VARIABLE_MAP.items()}
+    attributes = [amap.get(a, a) for a in attributes]
+    attributes = list(set(attributes))  # remove duplicate values
+
     # required query-string parameters for request to PSM3 API
     params = {
         'api_key': api_key,
@@ -167,12 +199,12 @@ def get_psm3(latitude, longitude, api_key, email, names='tmy', interval=60,
     # the CSV is in the response content as a UTF-8 bytestring
     # to use pandas we need to create a file buffer from the response
     fbuf = io.StringIO(response.content.decode('utf-8'))
-    return parse_psm3(fbuf)
+    return parse_psm3(fbuf, map_variables)
 
 
-def parse_psm3(fbuf):
+def parse_psm3(fbuf, map_variables=None):
     """
-    Parse an NSRDB PSM3 weather file (formatted as SAM CSV).  The NSRDB
+    Parse an NSRDB PSM3 weather file (formatted as SAM CSV). The NSRDB
     is described in [1]_ and the SAM CSV format is described in [2]_.
 
     .. versionchanged:: 0.9.0
@@ -184,6 +216,9 @@ def parse_psm3(fbuf):
     ----------
     fbuf: file-like object
         File-like object containing data to read.
+    map_variables: bool
+        When true, renames columns of the Dataframe to pvlib variable names
+        where applicable. See variable VARIABLE_MAP.
 
     Returns
     -------
@@ -296,12 +331,25 @@ def parse_psm3(fbuf):
     tz = 'Etc/GMT%+d' % -metadata['Time Zone']
     data.index = pd.DatetimeIndex(dtidx).tz_localize(tz)
 
+    if map_variables is None:
+        warnings.warn(
+            'PSM3 variable names will be renamed to pvlib conventions by '
+            'default starting in pvlib 0.11.0. Specify map_variables=True '
+            'to enable that behavior now, or specify map_variables=False '
+            'to hide this warning.', pvlibDeprecationWarning)
+        map_variables = False
+    if map_variables:
+        data = data.rename(columns=VARIABLE_MAP)
+        metadata['latitude'] = metadata.pop('Latitude')
+        metadata['longitude'] = metadata.pop('Longitude')
+        metadata['altitude'] = metadata.pop('Elevation')
+
     return data, metadata
 
 
-def read_psm3(filename):
+def read_psm3(filename, map_variables=None):
     """
-    Read an NSRDB PSM3 weather file (formatted as SAM CSV).  The NSRDB
+    Read an NSRDB PSM3 weather file (formatted as SAM CSV). The NSRDB
     is described in [1]_ and the SAM CSV format is described in [2]_.
 
     .. versionchanged:: 0.9.0
@@ -313,6 +361,9 @@ def read_psm3(filename):
     ----------
     filename: str
         Filename of a file containing data to read.
+    map_variables: bool
+        When true, renames columns of the Dataframe to pvlib variable names
+        where applicable. See variable VARIABLE_MAP.
 
     Returns
     -------
@@ -334,5 +385,5 @@ def read_psm3(filename):
        <https://web.archive.org/web/20170207203107/https://sam.nrel.gov/sites/default/files/content/documents/pdf/wfcsv.pdf>`_
     """
     with open(str(filename), 'r') as fbuf:
-        content = parse_psm3(fbuf)
+        content = parse_psm3(fbuf, map_variables)
     return content
diff --git a/pvlib/iotools/sodapro.py b/pvlib/iotools/sodapro.py
index 68fa82a..abb06da 100644
--- a/pvlib/iotools/sodapro.py
+++ b/pvlib/iotools/sodapro.py
@@ -55,10 +55,8 @@ def get_cams(latitude, longitude, start, end, email, identifier='mcclear',
     Access: free, but requires registration, see [1]_
 
     Requests: max. 100 per day
-
-    Geographical coverage: Wordwide for CAMS McClear and -66° to 66° in both
-    latitude and longitude for CAMS Radiation
-
+    Geographical coverage: worldwide for CAMS McClear and approximately -66° to
+    66° in both latitude and longitude for CAMS Radiation.
 
     Parameters
     ----------


Test patch:
diff --git a/pvlib/tests/iotools/test_psm3.py b/pvlib/tests/iotools/test_psm3.py
index 92451a2..2c20155 100644
--- a/pvlib/tests/iotools/test_psm3.py
+++ b/pvlib/tests/iotools/test_psm3.py
@@ -4,13 +4,14 @@ test iotools for PSM3
 
 import os
 from pvlib.iotools import psm3
-from ..conftest import DATA_DIR, RERUNS, RERUNS_DELAY
+from ..conftest import DATA_DIR, RERUNS, RERUNS_DELAY, assert_index_equal
 import numpy as np
 import pandas as pd
 import pytest
 from requests import HTTPError
 from io import StringIO
 import warnings
+from pvlib._deprecation import pvlibDeprecationWarning
 
 TMY_TEST_DATA = DATA_DIR / 'test_psm3_tmy-2017.csv'
 YEAR_TEST_DATA = DATA_DIR / 'test_psm3_2017.csv'
@@ -76,7 +77,8 @@ def assert_psm3_equal(data, metadata, expected):
 def test_get_psm3_tmy(nrel_api_key):
     """test get_psm3 with a TMY"""
     data, metadata = psm3.get_psm3(LATITUDE, LONGITUDE, nrel_api_key,
-                                   PVLIB_EMAIL, names='tmy-2017')
+                                   PVLIB_EMAIL, names='tmy-2017',
+                                   map_variables=False)
     expected = pd.read_csv(TMY_TEST_DATA)
     assert_psm3_equal(data, metadata, expected)
 
@@ -86,7 +88,8 @@ def test_get_psm3_tmy(nrel_api_key):
 def test_get_psm3_singleyear(nrel_api_key):
     """test get_psm3 with a single year"""
     data, metadata = psm3.get_psm3(LATITUDE, LONGITUDE, nrel_api_key,
-                                   PVLIB_EMAIL, names='2017', interval=30)
+                                   PVLIB_EMAIL, names='2017',
+                                   map_variables=False, interval=30)
     expected = pd.read_csv(YEAR_TEST_DATA)
     assert_psm3_equal(data, metadata, expected)
 
@@ -96,7 +99,8 @@ def test_get_psm3_singleyear(nrel_api_key):
 def test_get_psm3_5min(nrel_api_key):
     """test get_psm3 for 5-minute data"""
     data, metadata = psm3.get_psm3(LATITUDE, LONGITUDE, nrel_api_key,
-                                   PVLIB_EMAIL, names='2019', interval=5)
+                                   PVLIB_EMAIL, names='2019', interval=5,
+                                   map_variables=False)
     assert len(data) == 525600/5
     first_day = data.loc['2019-01-01']
     expected = pd.read_csv(YEAR_TEST_DATA_5MIN)
@@ -108,7 +112,7 @@ def test_get_psm3_5min(nrel_api_key):
 def test_get_psm3_check_leap_day(nrel_api_key):
     data_2012, _ = psm3.get_psm3(LATITUDE, LONGITUDE, nrel_api_key,
                                  PVLIB_EMAIL, names="2012", interval=60,
-                                 leap_day=True)
+                                 leap_day=True, map_variables=False)
     assert len(data_2012) == (8760 + 24)
 
 
@@ -133,7 +137,7 @@ def test_get_psm3_tmy_errors(
     """
     with pytest.raises(HTTPError) as excinfo:
         psm3.get_psm3(latitude, longitude, api_key, PVLIB_EMAIL,
-                      names=names, interval=interval)
+                      names=names, interval=interval, map_variables=False)
     # ensure the HTTPError caught isn't due to overuse of the API key
     assert "OVER_RATE_LIMIT" not in str(excinfo.value)
 
@@ -149,13 +153,49 @@ def io_input(request):
 
 def test_parse_psm3(io_input):
     """test parse_psm3"""
-    data, metadata = psm3.parse_psm3(io_input)
+    data, metadata = psm3.parse_psm3(io_input, map_variables=False)
     expected = pd.read_csv(YEAR_TEST_DATA)
     assert_psm3_equal(data, metadata, expected)
 
 
 def test_read_psm3():
     """test read_psm3"""
-    data, metadata = psm3.read_psm3(MANUAL_TEST_DATA)
+    data, metadata = psm3.read_psm3(MANUAL_TEST_DATA, map_variables=False)
     expected = pd.read_csv(YEAR_TEST_DATA)
     assert_psm3_equal(data, metadata, expected)
+
+
+def test_read_psm3_map_variables():
+    """test read_psm3 map_variables=True"""
+    data, metadata = psm3.read_psm3(MANUAL_TEST_DATA, map_variables=True)
+    columns_mapped = ['Year', 'Month', 'Day', 'Hour', 'Minute', 'dhi', 'dni',
+                      'ghi', 'dhi_clear', 'dni_clear', 'ghi_clear',
+                      'Cloud Type', 'Dew Point', 'apparent_zenith',
+                      'Fill Flag', 'albedo', 'wind_speed',
+                      'precipitable_water', 'wind_direction',
+                      'relative_humidity', 'temp_air', 'pressure']
+    data, metadata = psm3.read_psm3(MANUAL_TEST_DATA, map_variables=True)
+    assert_index_equal(data.columns, pd.Index(columns_mapped))
+
+
+@pytest.mark.remote_data
+@pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
+def test_get_psm3_attribute_mapping(nrel_api_key):
+    """Test that pvlib names can be passed in as attributes and get correctly
+    reverse mapped to PSM3 names"""
+    data, meta = psm3.get_psm3(LATITUDE, LONGITUDE, nrel_api_key, PVLIB_EMAIL,
+                               names=2019, interval=60,
+                               attributes=['ghi', 'wind_speed'],
+                               map_variables=True)
+    assert 'ghi' in data.columns
+    assert 'wind_speed' in data.columns
+    assert 'latitude' in meta.keys()
+    assert 'longitude' in meta.keys()
+    assert 'altitude' in meta.keys()
+
+
+@pytest.mark.remote_data
+@pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
+def test_psm3_variable_map_deprecation_warning(nrel_api_key):
+    with pytest.warns(pvlibDeprecationWarning, match='names will be renamed'):
+        _ = psm3.read_psm3(MANUAL_TEST_DATA)