Repository: openclimatefix/nowcasting_forecast (language: python)

Issue:
Solar forecast consistly over estimates
Over the last few days, solar forecast seems to be constantly over forecasting


One clue could be 
- plot all MAE's per GSP and see if an stick out?
- seems to be large errors around Sheffield? Is there a funny PV system here


Discussion / hints:
<img width="1338" alt="Screenshot 2022-05-30 at 12 41 07" src="https://user-images.githubusercontent.com/34686298/170985033-8eedb7dd-504f-43f2-9d4d-0ee976754394.png">

MAE over last day per GSP
![Uploading Screenshot 2022-05-30 at 12.48.01.png…]()

Also Keadby seems to at zero, for a long time
<img width="774" alt="Screenshot 2022-05-30 at 13 44 24" src="https://user-images.githubusercontent.com/34686298/170995040-1030f754-186f-45f0-83d9-54c8ae1b6099.png">

There system have pv yields over 100%. 

Actions: 34165 reduce capacity - update to 5.7 -[ pvoutput.org](https://pvoutput.org/intraday.jsp?id=34165&sid=34165&dt=20220530)
[24565](https://pvoutput.org/intraday.jsp?id=24565&sid=24565&dt=20220530) - 4kwp
[42795](https://pvoutput.org/intraday.jsp?id=42795&sid=42795&dt=20220530) - 4
[49643](https://pvoutput.org/intraday.jsp?id=49643&sid=49643&dt=20220530) - 3.5
[67123](https://pvoutput.org/intraday.jsp?id=67123&sid=67123&dt=20220530) - 10.2 OCF: 7.6
[63689](https://pvoutput.org/intraday.jsp?id=63689&sid=63689&dt=20220530) - 4.2

Other systems that might need checking
[58944](https://pvoutput.org/intraday.jsp?id=58944&sid=58944&dt=20220530) - 3 OCF database: 3.2
[47944](https://pvoutput.org/intraday.jsp?id=47944&sid=47944&dt=20220530) - 0.75, OCF database: 0.7
[41651](https://pvoutput.org/intraday.jsp?id=41651&sid=41651&dt=20220530) - 4, OCF database: 3.7
[31970](https://pvoutput.org/intraday.jsp?id=31970&sid=31970&dt=20220530) - 3.3 OCF database: 3.0
[21570](https://pvoutput.org/intraday.jsp?id=21570&sid=21570&dt=20220530) - 5.4 OCF database: 3.6
[13551](https://pvoutput.org/intraday.jsp?id=31970&sid=13551&dt=20220530) - 8.3: OCF database: 4.4

sql
```
select DISTINCT on (pv_system.pv_system_id) solar_generation_kw,installed_capacity_kw,pv_system.pv_system_id,  * from pv_yield
join pv_system on pv_yield.pv_system_id = pv_system.id
where datetime_utc > '2022-05-22'
and solar_generation_kw > installed_capacity_kw * 1.05
```
Decided to update them all to be max values
<img width="676" alt="Screenshot 2022-05-31 at 11 30 01" src="https://user-images.githubusercontent.com/34686298/171153697-1ac559da-6e12-4597-ac77-7c89615496cb.png">
Looks slightly better today, but not totally
Some plotting errors on main app - https://github.com/openclimatefix/nowcasting/issues/62
Looking at v15 data 0 batch, 0 example, we t0=09.15 and gsp times are:
array(['2021-08-06T07:00:00.000000000', '2021-08-06T07:30:00.000000000',
       '2021-08-06T08:00:00.000000000', '2021-08-06T08:30:00.000000000',
       '2021-08-06T09:00:00.000000000', '2021-08-06T09:30:00.000000000',
       '2021-08-06T10:00:00.000000000', '2021-08-06T10:30:00.000000000',
       '2021-08-06T11:00:00.000000000', '2021-08-06T11:30:00.000000000',
       '2021-08-06T12:00:00.000000000', '2021-08-06T12:30:00.000000000',
       '2021-08-06T13:00:00.000000000', '2021-08-06T13:30:00.000000000',
       '2021-08-06T14:00:00.000000000', '2021-08-06T14:30:00.000000000',
       '2021-08-06T15:00:00.000000000', '2021-08-06T15:30:00.000000000',
       '2021-08-06T16:00:00.000000000', '2021-08-06T16:30:00.000000000',
       '2021-08-06T17:00:00.000000000'], dtype='datetime64[ns]')

1st target time is 09.30

when looking at live we see
t0=20.00 and gso datetime sare
array(['2022-05-31T18:30:00.000000000', '2022-05-31T19:00:00.000000000',
       '2022-05-31T19:30:00.000000000', '2022-05-31T20:00:00.000000000',
       '2022-05-31T20:30:00.000000000'], dtype='datetime64[ns]')

1st target time is 20.30

So ....
Note in training t0 ahead of 5th data point by 15mins, but in live, t0 is equal of 5t datapoint by 30 mins. 
In live we should load data from 18:00 to 20:00. This mistake woulc cause an over predictoin in the morning and underprediction in the afternoon. 




Fix patch:
diff --git a/requirements.txt b/requirements.txt
index d79d2fe6..7c7328a0 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -3,7 +3,7 @@ sqlalchemy
 psycopg2-binary
 numpy
 click
-nowcasting_dataset==3.5.2
+nowcasting_dataset==3.5.3
 nowcasting_dataloader==1.8.38
 nowcasting_datamodel==1.0.9
 fsspec


Test patch:
diff --git a/tests/test_batch.py b/tests/test_batch.py
index 4fb0b2eb..1b3e2a95 100644
--- a/tests/test_batch.py
+++ b/tests/test_batch.py
@@ -1,14 +1,18 @@
 import os
 import tempfile
+from datetime import datetime, timezone
 
+import pandas as pd
 import xarray as xr
 import zarr
 from nowcasting_dataset.config.load import load_yaml_configuration
 from nowcasting_dataset.config.save import save_yaml_configuration
+from nowcasting_dataset.data_sources.gsp.gsp_model import GSP
 from nowcasting_dataset.data_sources.pv.pv_model import PV
 from nowcasting_dataset.data_sources.satellite.satellite_model import Satellite
 
 from nowcasting_forecast.batch import make_batches
+from nowcasting_forecast.utils import floor_minutes_dt
 
 
 def test_make_batches(nwp_data):
@@ -99,3 +103,11 @@ def test_make_batches_mvp_v2(
         pv = xr.load_dataset(f"{temp_dir}/live/pv/000000.nc", engine="h5netcdf")
         pv = PV(pv)
         assert pv.power_mw.max() > 0
+
+        gsp = xr.load_dataset(f"{temp_dir}/live/gsp/000000.nc", engine="h5netcdf")
+        gsp = GSP(gsp)
+        assert len(gsp.time.values[0]) == 5
+        assert (
+            pd.to_datetime(gsp.time.values[0, -1]).isoformat()
+            == floor_minutes_dt(datetime.now(tz=timezone.utc)).replace(tzinfo=None).isoformat()
+        )