Repository: pvlib/pvlib-python

Issue:
Bug within scaling.py wavelet calculation methodology
**Describe the bug**
Mathematical error within the wavelet computation for the scaling.py WVM implementation. Error arises from the methodology, as opposed to just a software bug. 

**To Reproduce**
Steps to reproduce the behavior:
```
import numpy as np
from pvlib import scaling
cs = np.random.rand(2**14)
w, ts = scaling._compute_wavelet(cs,1)
print(np.all( (sum(w)-cs) < 1e-8 ))  # Returns False, expect True
```

**Expected behavior**
For a discrete wavelet transform (DWT) the sum of all wavelet modes should equate to the original data. 

**Versions:**
 - ``pvlib.__version__``: 0.7.2
 - ``pandas.__version__``: 1.2.3
 - python: 3.8.8

**Additional context**
This bug is also present in the [PV_LIB](https://pvpmc.sandia.gov/applications/wavelet-variability-model/) Matlab version that was used as the basis for this code (I did reach out to them using the PVLIB MATLAB email form, but don't know who actually wrote that code). Essentially, the existing code throws away the highest level of Detail Coefficient in the transform and keeps an extra level of Approximation coefficient. The impact on the calculation is small, but leads to an incorrect DWT and reconstruction. I have a fix that makes the code pass the theoretical test about the DWT proposed under 'To Reproduce' but there may be some question as to whether this should be corrected or left alone to match the MATLAB code it was based on. 



Discussion:
@jranalli thanks for finding and reporting this. Can I ask how you contacted PVLIB MATLAB? Because I maintain that repository and I didn't see the email, so we need to fix something on our end with communications.
@cwhanse Now that I look again, I think I used the wrong form. It was just a general Questions and Comments link for the PV Performance Modeling Collaborative at the bottom of the page. I didn't see any contact point for the PV_LIB MATLAB library and I also didn't know about the github repo for it, but now I do! 

I do have a fix for the MATLAB code as well, but I don't see that part of the library on github. If you'd like me to open an issue on that repository as well, I'd be happy to do so, but if there's some other pathway or contact point since that's kind of listed as a separate package of the code, please let me know.

Either way, do you think it's appropriate to fix this, or does there need to be a conversation with the original author of that MATLAB code? If everything is fine to go ahead with it here, I can just put together my fix as a pull request for review.


And did my own looking: pvl_WVM is in it's own Matlab archive, separate from PVLIB for Matlab. The WVM code is only available as a download from pvpmc.sandia.gov, whereas PVLIB for Matlab is on [github](https://github.com/sandialabs/MATLAB_PV_LIB).

I've sent the bug report to Matt Lave, the originator of the WVM algorithm and code. We'll likely welcome the bug fix but I'd like to hear Matt's view first.
OK sounds good. If he or you want to connect for more detail on the issue, you can get contact info for me at my [Faculty Page](http://personal.psu.edu/jar339/about.html).
for the record: bug is confirmed via separate communication with the WVM algorithm author.

Patch:
diff --git a/pvlib/scaling.py b/pvlib/scaling.py
--- a/pvlib/scaling.py
+++ b/pvlib/scaling.py
@@ -13,8 +13,8 @@
 def wvm(clearsky_index, positions, cloud_speed, dt=None):
     """
     Compute spatial aggregation time series smoothing on clear sky index based
-    on the Wavelet Variability model of Lave et al [1-2]. Implementation is
-    basically a port of the Matlab version of the code [3].
+    on the Wavelet Variability model of Lave et al. [1]_, [2]_. Implementation
+    is basically a port of the Matlab version of the code [3]_.
 
     Parameters
     ----------
@@ -48,16 +48,16 @@ def wvm(clearsky_index, positions, cloud_speed, dt=None):
 
     References
     ----------
-    [1] M. Lave, J. Kleissl and J.S. Stein. A Wavelet-Based Variability
-    Model (WVM) for Solar PV Power Plants. IEEE Transactions on Sustainable
-    Energy, vol. 4, no. 2, pp. 501-509, 2013.
+    .. [1] M. Lave, J. Kleissl and J.S. Stein. A Wavelet-Based Variability
+       Model (WVM) for Solar PV Power Plants. IEEE Transactions on Sustainable
+       Energy, vol. 4, no. 2, pp. 501-509, 2013.
 
-    [2] M. Lave and J. Kleissl. Cloud speed impact on solar variability
-    scaling - Application to the wavelet variability model. Solar Energy,
-    vol. 91, pp. 11-21, 2013.
+    .. [2] M. Lave and J. Kleissl. Cloud speed impact on solar variability
+       scaling - Application to the wavelet variability model. Solar Energy,
+       vol. 91, pp. 11-21, 2013.
 
-    [3] Wavelet Variability Model - Matlab Code:
-    https://pvpmc.sandia.gov/applications/wavelet-variability-model/
+    .. [3] Wavelet Variability Model - Matlab Code:
+       https://github.com/sandialabs/wvm
     """
 
     # Added by Joe Ranalli (@jranalli), Penn State Hazleton, 2019
@@ -128,13 +128,13 @@ def latlon_to_xy(coordinates):
 
     References
     ----------
-    [1] H. Moritz. Geodetic Reference System 1980, Journal of Geodesy, vol. 74,
-    no. 1, pp 128–133, 2000.
+    .. [1] H. Moritz. Geodetic Reference System 1980, Journal of Geodesy, vol.
+       74, no. 1, pp 128–133, 2000.
 
-    [2] https://pypi.org/project/pyproj/
+    .. [2] https://pypi.org/project/pyproj/
 
-    [3] Wavelet Variability Model - Matlab Code:
-    https://pvpmc.sandia.gov/applications/wavelet-variability-model/
+    .. [3] Wavelet Variability Model - Matlab Code:
+       https://github.com/sandialabs/wvm
     """
 
     # Added by Joe Ranalli (@jranalli), Penn State Hazleton, 2019
@@ -159,7 +159,12 @@ def latlon_to_xy(coordinates):
 
 def _compute_wavelet(clearsky_index, dt=None):
     """
-    Compute the wavelet transform on the input clear_sky time series.
+    Compute the wavelet transform on the input clear_sky time series. Uses a
+    top hat wavelet [-1,1,1,-1] shape, based on the difference of successive
+    centered moving averages. Smallest scale (filter size of 2) is a degenerate
+    case that resembles a Haar wavelet. Returns one level of approximation
+    coefficient (CAn) and n levels of detail coefficients (CD1, CD2, ...,
+    CDn-1, CDn).
 
     Parameters
     ----------
@@ -174,19 +179,20 @@ def _compute_wavelet(clearsky_index, dt=None):
     Returns
     -------
     wavelet: numeric
-        The individual wavelets for the time series
+        The individual wavelets for the time series. Format follows increasing
+        scale (decreasing frequency): [CD1, CD2, ..., CDn, CAn]
 
     tmscales: numeric
         The timescales associated with the wavelets in seconds [s]
 
     References
     ----------
-    [1] M. Lave, J. Kleissl and J.S. Stein. A Wavelet-Based Variability
-    Model (WVM) for Solar PV Power Plants. IEEE Transactions on Sustainable
-    Energy, vol. 4, no. 2, pp. 501-509, 2013.
+    .. [1] M. Lave, J. Kleissl and J.S. Stein. A Wavelet-Based Variability
+       Model (WVM) for Solar PV Power Plants. IEEE Transactions on
+       Sustainable Energy, vol. 4, no. 2, pp. 501-509, 2013.
 
-    [3] Wavelet Variability Model - Matlab Code:
-    https://pvpmc.sandia.gov/applications/wavelet-variability-model/
+    .. [2] Wavelet Variability Model - Matlab Code:
+       https://github.com/sandialabs/wvm
     """
 
     # Added by Joe Ranalli (@jranalli), Penn State Hazleton, 2019
@@ -209,31 +215,37 @@ def _compute_wavelet(clearsky_index, dt=None):
 
     # Compute wavelet time scales
     min_tmscale = np.ceil(np.log(dt)/np.log(2))  # Minimum wavelet timescale
-    max_tmscale = int(12 - min_tmscale)  # maximum wavelet timescale
+    max_tmscale = int(13 - min_tmscale)  # maximum wavelet timescale
 
     tmscales = np.zeros(max_tmscale)
     csi_mean = np.zeros([max_tmscale, len(cs_long)])
+    # Skip averaging for the 0th scale
+    csi_mean[0, :] = cs_long.values.flatten()
+    tmscales[0] = 1
     # Loop for all time scales we will consider
-    for i in np.arange(0, max_tmscale):
-        j = i+1
-        tmscales[i] = 2**j * dt  # Wavelet integration time scale
-        intvlen = 2**j  # Wavelet integration time series interval
+    for i in np.arange(1, max_tmscale):
+        tmscales[i] = 2**i * dt  # Wavelet integration time scale
+        intvlen = 2**i  # Wavelet integration time series interval
         # Rolling average, retains only lower frequencies than interval
+        # Produces slightly different end effects than the MATLAB version
         df = cs_long.rolling(window=intvlen, center=True, min_periods=1).mean()
         # Fill nan's in both directions
         df = df.fillna(method='bfill').fillna(method='ffill')
         # Pop values back out of the dataframe and store
         csi_mean[i, :] = df.values.flatten()
+        # Shift to account for different indexing in MATLAB moving average
+        csi_mean[i, :] = np.roll(csi_mean[i, :], -1)
+        csi_mean[i, -1] = csi_mean[i, -2]
 
-    # Calculate the wavelets by isolating the rolling mean frequency ranges
+    # Calculate detail coefficients by difference between successive averages
     wavelet_long = np.zeros(csi_mean.shape)
     for i in np.arange(0, max_tmscale-1):
         wavelet_long[i, :] = csi_mean[i, :] - csi_mean[i+1, :]
-    wavelet_long[max_tmscale-1, :] = csi_mean[max_tmscale-1, :]  # Lowest freq
+    wavelet_long[-1, :] = csi_mean[-1, :]  # Lowest freq (CAn)
 
     # Clip off the padding and just return the original time window
     wavelet = np.zeros([max_tmscale, len(vals)])
     for i in np.arange(0, max_tmscale):
-        wavelet[i, :] = wavelet_long[i, len(vals)+1: 2*len(vals)+1]
+        wavelet[i, :] = wavelet_long[i, len(vals): 2*len(vals)]
 
     return wavelet, tmscales


Test patch:
diff --git a/pvlib/tests/test_scaling.py b/pvlib/tests/test_scaling.py
--- a/pvlib/tests/test_scaling.py
+++ b/pvlib/tests/test_scaling.py
@@ -48,21 +48,24 @@ def positions():
 @pytest.fixture
 def expect_tmscale():
     # Expected timescales for dt = 1
-    return [2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096]
+    return [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096]
 
 
 @pytest.fixture
 def expect_wavelet():
     # Expected wavelet for indices 5000:5004 for clear_sky_index above (Matlab)
-    return np.array([[-0.025, 0.05, 0., -0.05, 0.025],
-                     [0.025, 0., 0., 0., -0.025],
-                     [0., 0., 0., 0., 0.]])
+    e = np.zeros([13, 5])
+    e[0, :] = np.array([0, -0.05, 0.1, -0.05, 0])
+    e[1, :] = np.array([-0.025, 0.05, 0., -0.05, 0.025])
+    e[2, :] = np.array([0.025, 0., 0., 0., -0.025])
+    e[-1, :] = np.array([1, 1, 1, 1, 1])
+    return e
 
 
 @pytest.fixture
 def expect_cs_smooth():
     # Expected smoothed clear sky index for indices 5000:5004 (Matlab)
-    return np.array([1., 1.0289, 1., 0.9711, 1.])
+    return np.array([1., 1., 1.05774, 0.94226, 1.])
 
 
 def test_latlon_to_xy_zero():
@@ -94,7 +97,7 @@ def test_compute_wavelet_series(clear_sky_index, time,
     csi_series = pd.Series(clear_sky_index, index=time)
     wavelet, tmscale = scaling._compute_wavelet(csi_series)
     assert_almost_equal(tmscale, expect_tmscale)
-    assert_almost_equal(wavelet[0:3, 5000:5005], expect_wavelet)
+    assert_almost_equal(wavelet[:, 5000:5005], expect_wavelet)
 
 
 def test_compute_wavelet_series_numindex(clear_sky_index, time,
@@ -103,14 +106,14 @@ def test_compute_wavelet_series_numindex(clear_sky_index, time,
     csi_series = pd.Series(clear_sky_index, index=dtindex)
     wavelet, tmscale = scaling._compute_wavelet(csi_series)
     assert_almost_equal(tmscale, expect_tmscale)
-    assert_almost_equal(wavelet[0:3, 5000:5005], expect_wavelet)
+    assert_almost_equal(wavelet[:, 5000:5005], expect_wavelet)
 
 
 def test_compute_wavelet_array(clear_sky_index,
                                expect_tmscale, expect_wavelet):
     wavelet, tmscale = scaling._compute_wavelet(clear_sky_index, dt)
     assert_almost_equal(tmscale, expect_tmscale)
-    assert_almost_equal(wavelet[0:3, 5000:5005], expect_wavelet)
+    assert_almost_equal(wavelet[:, 5000:5005], expect_wavelet)
 
 
 def test_compute_wavelet_array_invalid(clear_sky_index):
@@ -118,6 +121,14 @@ def test_compute_wavelet_array_invalid(clear_sky_index):
         scaling._compute_wavelet(clear_sky_index)
 
 
+def test_compute_wavelet_dwttheory(clear_sky_index, time,
+                                   expect_tmscale, expect_wavelet):
+    # Confirm detail coeffs sum to original signal
+    csi_series = pd.Series(clear_sky_index, index=time)
+    wavelet, tmscale = scaling._compute_wavelet(csi_series)
+    assert_almost_equal(np.sum(wavelet, 0), csi_series)
+
+
 def test_wvm_series(clear_sky_index, time, positions, expect_cs_smooth):
     csi_series = pd.Series(clear_sky_index, index=time)
     cs_sm, _, _ = scaling.wvm(csi_series, positions, cloud_speed)