Repository: pvlib/pvlib-python

Issue:
deprecate existing code in forecast.py, possibly replace with solarforecastarbiter shim
`forecast.py` is a burden to maintain. I haven't used it in years, I don't think any of the other pvlib maintainers are interested in it, and I don't see any users stepping up to volunteer to maintain it. The code is not up to my present standards and I don't see how I'd get it there without a complete rewrite. This leads to difficult to track bugs such as the one recently reported on the [google group](https://groups.google.com/g/pvlib-python/c/b9HdgWV6w6g). It also complicates the pvlib dependencies.

[solarforecastarbiter](https://github.com/SolarArbiter/solarforecastarbiter-core) includes a [reference_forecasts](https://github.com/SolarArbiter/solarforecastarbiter-core/tree/master/solarforecastarbiter/reference_forecasts) package that is much more robust. See [documentation here](https://solarforecastarbiter-core.readthedocs.io/en/latest/reference-forecasts.html) and [example notebook here](https://github.com/SolarArbiter/workshop/blob/master/reference_forecasts.ipynb) (no promises that this works without modification for the latest version).

The main reason to prefer `forecast.py` to `solarforecastarbiter` is the data fetch process. `forecast.py` pulls point data from a Unidata THREDDS server. `solarforecastarbiter.reference_forecasts` assumes you already have gridded data stored in a netcdf file. `solarforecastarbiter.io.nwp` provides functions to fetch that gridded data from NCEP. We have very good reasons for that approach in `solarforecastarbiter`, but I doubt that many `forecast.py` users are interested in configuring that two step process for their application.

I'm very tempted to stop here, remove `forecast.py` after deprecation, and say "not my problem anymore", but it seems to attract a fair number of people to pvlib, so I hesitate to remove it without some kind of replacement. Let's explore a few ideas.

1. Within `forecast.py`, rewrite code to fetch relevant data from Unidata. Make this function compatible with the expectations for the [`load_forecast`](https://github.com/SolarArbiter/solarforecastarbiter-core/blob/6200ec067bf83bc198a3af59da1d924d4124d4ec/solarforecastarbiter/reference_forecasts/models.py#L16-L19) function passed into `solarforecastarbiter.reference_forecasts.models` functions.
2. Same as 1., except put that code somewhere else. Could be a documentation example, could be in solarforecastarbiter, or could be in a gist.
3. Copy/refactor solarforecastarbiter code into `forecast.py`.
4. Do nothing and let the forecast.py bugs and technical debt pile up. 

Other thoughts?


Discussion:
I used these forecast functions earlier in my career for production forecasting (getting forecasted irradiance data). But that said, everything I used it for can be done with different tools (some already in pvlib). There are some good free/paid weather services out there that return weather forecast data, and pvlib already has functions to determine irradiance from cloud_coverage.  

While it could be tough for some to deprecate `forecast.py`, you have other tools that provide solutions and inputs. I have no problem with it being removed, especially if the amount of work to maintain is greater than the number of users who utilize it.
Probably not a solution but I recently started a weather data service to more easily access time-series reanalysis (ERA5) and forecast data (GFS) (see example here - https://oikolab.com/documentation). It's currently going through beta testing but I've been thinking about how to offer some of the service to the open-source / academic community. 

If you have any suggestion, would be more than happy to discuss.
I've looked at the `forecast.py` code and I'm here to report a bug on it, so yeah, I see the problem.

I count myself as a newbie user attracted to pvlib in part because of the forecasting. Even so I have no problem with it going away, as long as we have some examples of how to use a different tool for forecasting, if only to be pointed at the existence of the other tool.

Regarding the ideas posted by @wholmgren. If you have no time to work on it, (4) is your only option. If not, you're clearly not in love with `forecast.py` ("I have no problem in it going away"), so I suggest you work toward deprecation and removal. For that, (2) seems the best approach. I also think (2) is the best approach to promote "separation of concerns": Getting and formatting the data seems like a useful piece of functionality by itself so it would be good if it existed as such, rather than buried in the `ForecastModel` class.
I’ve used `forecast.py` and have found it to be the most straightforward way to get data for pv production forecasting.
I played around with `solarforecastarbiter` for around a day.  There appear to be lots of interesting functionality to it but I found the expectation of having externally downloaded data outside of the library’s API stored locally a hurdle to using it. Maybe I didn’t give it the chance it deserved or maybe this is something a documentation example can/does solve, but I’m just echoing your point that part of the allure of `forecast.py` is that it pulls the data for you from an external source, eliminating the need to deal with file management yourself, and allows you to stay within pvlib the whole time.

A complete and simple forecasting example within pvlib is a powerful use case for the library.  All that said, If the example uses some “lean” code from `solarforecastarbiter` that’s probably fine too.
possibly of interest as an alternative data source: https://azure.microsoft.com/en-us/services/open-datasets/catalog/noaa-global-forecast-system/
After a year and half of mostly following option 4 (`Do nothing and let the forecast.py bugs and technical debt pile up.`), and another release around the corner, maybe it's time to restart this discussion.  @wholmgren, has the arbiter evolved in a way that changes any of your thoughts at the top of this thread?

If the data fetching code needs to live on somewhere, maybe `pvlib.iotools` is as good a place as any.  However I'm not sure that will relieve the maintenance burden much -- of the dozen or so forecast issues opened in the last year, seems like the majority have to do with fetching data.  The dependency complication is for the data fetching as well.
My initial reaction to every `forecast.py` issue remains "remove it." Short of that, putting the fetch in `pvlib.iotools` and copying SFA's [`forecast.py`](https://github.com/SolarArbiter/solarforecastarbiter-core/blob/master/solarforecastarbiter/reference_forecasts/forecast.py) (either as a whole module or putting contents in other places like irradiance.py) feels like the best approach. I have no time or interest to maintain the original or any refactored code.
I also have no interest in maintaining the original, maintaining a refactored version, or doing the refactor itself.  I would be willing to deprecate the contents of `forecast.py` because it's easy and means less maintenance in the long run.

Any objections to deprecating without replacement?  Any objection to including the deprecation in 0.9.1?
My vote would be to move the code to a its own package but I'm with @wholmgren and @kanderso-nrel I don't have the bandwidth to maintain another project.  Maybe we can canvas the user community for volunteers. I support deprecating `forecast.py` perhaps that will encourage someone to come forward to pick it up.

Patch:
diff --git a/pvlib/forecast.py b/pvlib/forecast.py
--- a/pvlib/forecast.py
+++ b/pvlib/forecast.py
@@ -15,6 +15,7 @@
 from siphon.ncss import NCSS
 
 import warnings
+from pvlib._deprecation import deprecated
 
 
 warnings.warn(
@@ -22,7 +23,15 @@
     'The API may change, the functionality may be consolidated into an io '
     'module, or the module may be separated into its own package.')
 
+_forecast_deprecated = deprecated(
+    since='0.9.1',
+    removal='a future release',
+    addendum='For details, see https://pvlib-python.readthedocs.io/en/stable/user_guide/forecasts.html'  # noqa: E501
+)
 
+# don't decorate the base class to prevent the subclasses from showing
+# duplicate warnings:
+# @_forecast_deprecated
 class ForecastModel:
     """
     An object for querying and holding forecast model information for
@@ -684,6 +693,7 @@ def gust_to_speed(self, data, scaling=1/1.4):
         return wind_speed
 
 
+@_forecast_deprecated
 class GFS(ForecastModel):
     """
     Subclass of the ForecastModel class representing GFS
@@ -785,6 +795,7 @@ def process_data(self, data, cloud_cover='total_clouds', **kwargs):
         return data[self.output_variables]
 
 
+@_forecast_deprecated
 class HRRR_ESRL(ForecastModel):                                 # noqa: N801
     """
     Subclass of the ForecastModel class representing
@@ -875,6 +886,7 @@ def process_data(self, data, cloud_cover='total_clouds', **kwargs):
         return data[self.output_variables]
 
 
+@_forecast_deprecated
 class NAM(ForecastModel):
     """
     Subclass of the ForecastModel class representing NAM
@@ -956,6 +968,7 @@ def process_data(self, data, cloud_cover='total_clouds', **kwargs):
         return data[self.output_variables]
 
 
+@_forecast_deprecated
 class HRRR(ForecastModel):
     """
     Subclass of the ForecastModel class representing HRRR
@@ -1044,6 +1057,7 @@ def process_data(self, data, cloud_cover='total_clouds', **kwargs):
         return data[self.output_variables]
 
 
+@_forecast_deprecated
 class NDFD(ForecastModel):
     """
     Subclass of the ForecastModel class representing NDFD forecast
@@ -1112,6 +1126,7 @@ def process_data(self, data, **kwargs):
         return data[self.output_variables]
 
 
+@_forecast_deprecated
 class RAP(ForecastModel):
     """
     Subclass of the ForecastModel class representing RAP forecast model.


Test patch:
diff --git a/pvlib/tests/test_forecast.py b/pvlib/tests/test_forecast.py
--- a/pvlib/tests/test_forecast.py
+++ b/pvlib/tests/test_forecast.py
@@ -14,6 +14,8 @@
 )
 from .conftest import RERUNS, RERUNS_DELAY
 
+from pvlib._deprecation import pvlibDeprecationWarning
+
 pytestmark = pytest.mark.skipif(not has_siphon, reason='requires siphon')
 
 
@@ -52,7 +54,8 @@
 @requires_siphon
 @pytest.fixture(scope='module', params=_modelclasses)
 def model(request):
-    amodel = request.param()
+    with pytest.warns(pvlibDeprecationWarning):
+        amodel = request.param()
     try:
         raw_data = amodel.get_data(_latitude, _longitude, _start, _end)
     except Exception as e:
@@ -90,7 +93,8 @@ def test_process_data(model):
 def test_bad_kwarg_get_data():
     # For more information on why you would want to pass an unknown keyword
     # argument, see Github issue #745.
-    amodel = NAM()
+    with pytest.warns(pvlibDeprecationWarning):
+        amodel = NAM()
     data = amodel.get_data(_latitude, _longitude, _start, _end,
                            bad_kwarg=False)
     assert not data.empty
@@ -103,7 +107,8 @@ def test_bad_kwarg_get_data():
 def test_bad_kwarg_get_processed_data():
     # For more information on why you would want to pass an unknown keyword
     # argument, see Github issue #745.
-    amodel = NAM()
+    with pytest.warns(pvlibDeprecationWarning):
+        amodel = NAM()
     data = amodel.get_processed_data(_latitude, _longitude, _start, _end,
                                      bad_kwarg=False)
     assert not data.empty
@@ -114,7 +119,8 @@ def test_bad_kwarg_get_processed_data():
 @pytest.mark.remote_data
 @pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
 def test_how_kwarg_get_processed_data():
-    amodel = NAM()
+    with pytest.warns(pvlibDeprecationWarning):
+        amodel = NAM()
     data = amodel.get_processed_data(_latitude, _longitude, _start, _end,
                                      how='clearsky_scaling')
     assert not data.empty
@@ -125,7 +131,8 @@ def test_how_kwarg_get_processed_data():
 @pytest.mark.remote_data
 @pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
 def test_vert_level():
-    amodel = NAM()
+    with pytest.warns(pvlibDeprecationWarning):
+        amodel = NAM()
     vert_level = 5000
     amodel.get_processed_data(_latitude, _longitude, _start, _end,
                               vert_level=vert_level)
@@ -136,7 +143,8 @@ def test_vert_level():
 @pytest.mark.remote_data
 @pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
 def test_datetime():
-    amodel = NAM()
+    with pytest.warns(pvlibDeprecationWarning):
+        amodel = NAM()
     start = datetime.now(tz=timezone.utc)
     end = start + timedelta(days=1)
     amodel.get_processed_data(_latitude, _longitude, start, end)
@@ -147,7 +155,8 @@ def test_datetime():
 @pytest.mark.remote_data
 @pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
 def test_queryvariables():
-    amodel = GFS()
+    with pytest.warns(pvlibDeprecationWarning):
+        amodel = GFS()
     new_variables = ['u-component_of_wind_height_above_ground']
     data = amodel.get_data(_latitude, _longitude, _start, _end,
                            query_variables=new_variables)
@@ -156,16 +165,19 @@ def test_queryvariables():
 
 @requires_siphon
 def test_latest():
-    GFS(set_type='latest')
+    with pytest.warns(pvlibDeprecationWarning):
+        GFS(set_type='latest')
 
 
 @requires_siphon
 def test_full():
-    GFS(set_type='full')
+    with pytest.warns(pvlibDeprecationWarning):
+        GFS(set_type='full')
 
 
 def test_temp_convert():
-    amodel = GFS()
+    with pytest.warns(pvlibDeprecationWarning):
+        amodel = GFS()
     data = pd.DataFrame({'temp_air': [273.15]})
     data['temp_air'] = amodel.kelvin_to_celsius(data['temp_air'])
 
@@ -183,27 +195,31 @@ def test_temp_convert():
 
 
 def test_set_location():
-    amodel = GFS()
+    with pytest.warns(pvlibDeprecationWarning):
+        amodel = GFS()
     latitude, longitude = 32.2, -110.9
     time = 'UTC'
     amodel.set_location(time, latitude, longitude)
 
 
 def test_set_query_time_range_tzfail():
-    amodel = GFS()
+    with pytest.warns(pvlibDeprecationWarning):
+        amodel = GFS()
     with pytest.raises(TypeError):
         amodel.set_query_time_range(datetime.now(), datetime.now())
 
 
 def test_cloud_cover_to_transmittance_linear():
-    amodel = GFS()
+    with pytest.warns(pvlibDeprecationWarning):
+        amodel = GFS()
     assert_allclose(amodel.cloud_cover_to_transmittance_linear(0), 0.75)
     assert_allclose(amodel.cloud_cover_to_transmittance_linear(100), 0.0)
     assert_allclose(amodel.cloud_cover_to_transmittance_linear(0, 0.5), 0.5)
 
 
 def test_cloud_cover_to_ghi_linear():
-    amodel = GFS()
+    with pytest.warns(pvlibDeprecationWarning):
+        amodel = GFS()
     ghi_clear = 1000
     offset = 25
     out = amodel.cloud_cover_to_ghi_linear(0, ghi_clear, offset=offset)