Repository: openclimatefix/nowcasting_forecast (language: python)

Issue:
No PV data in live batches
We see no (or very little PV data in the batches)

This might effect how the model is run, as when training, there are not empty fields



Discussion / hints:
Occasioanly do see results in the reight place, this might be when some PV data is available
<img width="398" alt="Screenshot 2022-05-19 at 13 30 16" src="https://user-images.githubusercontent.com/34686298/169293640-d90b72b7-679b-43e5-bfbe-cc3de4b250a2.png">


Hmm, lack of PV data is definitely an issue! Do we think this is a bug in our code? Or does it take ages for live PV data to flow through PVOutput.org?

Does our code discard PV data if any single timestep is missing? `power_perceiver` should be able to cope with missing timesteps in the PV data, and it will probably still really appreciate having any (sane) PV data that's available, even if it's missing a few timesteps :slightly_smiling_face: 
Ill have to check, my thinking right now is its just slower to get to us. I have noticed some sites just dont have data, see PV tab on[ data-vis](http://nowcasting-data-visualization-development.eu-west-1.elasticbeanstalk.com/)

One idea could be to load 1 hour of data, and interpolate forward, and then just use the interpolated amounts. 

Note the CNN model uses 30 mins of PV data currernly
For `power_perceiver`, please can we turn off all interpolation? `power_perceiver` expects the "raw" (normalised) data, and it can handle PV data at irregular time intervals.

And, yeah, having some "old" PV data would be far preferable to having no PV data :slightly_smiling_face:  e.g. let's say we're creating a forecast at 10:00am. And there's no PV data for 9:30am - 10:00am (because PV data takes a while to flow through PVOutput.org). Then it'd be great to be able to give `power_perceiver` whatever PV data we have for 8:30am-9:30am.

(BTW, if neither PVOutput.org nor Passiv can give us timely PV data, then that feels like a strong argument to build our own data collection system as part of the SBRI project! Getting PV data within seconds of the hardware taking the reading might turn out to be quite important).
We can those options in, intperolate=True
And yea this would be solved if we took a longer history of data. 

I think for this issue we should
1. Load 1.5 hours of pv history
2. interpolate up to now
3. only keep 30 mins of history

We should of course keep things flexible for `power_perciever` but I think right now, we should focus on getting resaonable outputs from CNN.


SGTM!
Another thing we could do is search for some other active sites. This might be a bigger job fro training, but I think this CNN doesnt mind too much
> Another thing we could do is search for some other active sites.

Please remind me: When does the code currently update its list of "active" PV sites? (I think I remember suggesting that, for now, we should update the list manually?)
They are hard coded
https://github.com/openclimatefix/PVConsumer/tree/main/pvconsumer/data

We can just update them and deploy a new version. 

we could ofcourse make this more dyanmic, but i think this is ok for the moment

Fix patch:
diff --git a/nowcasting_forecast/config/mvp_v2.yaml b/nowcasting_forecast/config/mvp_v2.yaml
index 4234a94f..7c71c42c 100644
--- a/nowcasting_forecast/config/mvp_v2.yaml
+++ b/nowcasting_forecast/config/mvp_v2.yaml
@@ -57,6 +57,8 @@ input_data:
   #--------- PV --------
   pv:
     is_live: True
+    live_interpolate_minutes: 60
+    live_load_extra_minutes: 30
 
 output_data:
   filepath: ./batches/
diff --git a/requirements.txt b/requirements.txt
index 0ad785c9..88ea471e 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -3,7 +3,7 @@ sqlalchemy
 psycopg2-binary
 numpy
 click
-nowcasting_dataset==3.3.14
+nowcasting_dataset==3.3.19
 nowcasting_dataloader==1.8.38
 nowcasting_datamodel==0.0.53
 fsspec


Test patch:
diff --git a/tests/models/cnn/test_run.py b/tests/models/cnn/test_run.py
index 70ceaca4..47f5e52d 100644
--- a/tests/models/cnn/test_run.py
+++ b/tests/models/cnn/test_run.py
@@ -3,7 +3,6 @@
 import os
 import tempfile
 
-import xarray as xr
 import zarr
 
 import nowcasting_forecast
diff --git a/tests/test_batch.py b/tests/test_batch.py
index 117ecae6..2bb7a813 100644
--- a/tests/test_batch.py
+++ b/tests/test_batch.py
@@ -79,6 +79,7 @@ def test_make_batches_mvp_v2(nwp_data, pv_yields_and_systems, sat_data, hrv_sat_
             sat_data.to_zarr(store, compute=True)
         os.environ["SAT_PATH"] = sat_path
 
+
         make_batches(
             config_filename="nowcasting_forecast/config/mvp_v2.yaml", temporary_dir=temp_dir
         )