Repository: openclimatefix/nowcasting_forecast (language: python)

Issue:
Investigate gap in forecast
2020-06-15th the forecast didnt seem to run in the night? 
why?

<img width="591" alt="Screenshot 2022-06-16 at 07 52 08" src="https://user-images.githubusercontent.com/34686298/174009730-fb581048-383e-4651-a910-b316af871b80.png">




Discussion / hints:
Happened again on 29th of June 

My suspicion is there is no PV data, as pvoutput.org seems to not give 0s in the night
Good catch!

> My suspicion is there is no PV data, as pvoutput.org seems to not give 0s in the night

Yes, a lot of PV systems (maybe _all_ PV systems, I'm not sure?) don't bother reporting _any_ data at night. But that should be true _every_ night (just not the night of the 29th June) :slightly_smiling_face: Or have I missed something?

On a related point, do we have a mechanism to allow us to backwards-fill the forecast? e.g. if the forecast doesn't run from midnight to 6am because something crashes, then, at 6am, can we back-fill the forecasts? (If not, let's not worry about this for WP2!)
**Backfill**
We do not have the ability to backfill right now. The process is defined quite specifically to be data efficient and not do that. 
Perhaps something for WP3, need to work out how much we need it

**Pv night**
Yea, I agree, perhaps some nights it just reaches over (so there is no gap in the yellow line), but yea I assume to this is the case. 

Solutions
1. Passiv data gives 0's in the night which is nice, so by adding passiv data the forecast could be find. This seems a bit risky and other things might need fixing. Like 
2. Add 0's to pv data, if elevation < 0. This seems a bit dodgy as we are adding data that is not there. Although it does seem a safe edit.
3. When loading pv data to nowcastin_dataset, if with elevation<0, then add some zeros. This should help smooth it over

My hunch is to go for 3. What do you think @JackKelly ?




**Backfill**
> Perhaps something for WP3, need to work out how much we need it

Good question. I'll add it to the list of things to discuss next week :slightly_smiling_face: 

**PV at night**
For now, appending zeros when the Sun is below the horizon sounds sensible! Although, are we sure PV systems don't generate during dawn and dusk (i.e. when the Sun is just below the horizon, so there's no _direct_ sunlight, but it's not "dark" because the sunlight is still scattering around the atmosphere).

In the near future, Power Perceiver should be able to handle "missing" PV data. i.e. if the PV data is NaN, then we shouldn't fill it with zeros or interpolate it at all, we just give the model the raw PV data, and tell the model "data for this PV system is missing at these timesteps". :slightly_smiling_face: 

**Broken forecast on 15th and 29th June**
Although, going back to the original topic of this thread (the forecast not running), are we _sure_ the forecast broke due to missing PV data? My hunch is still that the profile of the missing PV data should be roughly the same every night. So, if the system couldn't handle missing PV data, then I'd expect it to fail _every_ night?
**PV at night**

Perhaps we can do below -10, to be sure. But it feels right even to do 0

Thats good that in the near future PP can support that. Thats good. 


**Broken forecast on 15th and 29th June**
It looks like it does happen every night. 

Here's a image of the error. Something to do with size of things. 
Perhaps this a quick fix, and we dont need to start adding zeros. Ill have to look at it further
<img width="1104" alt="Screenshot 2022-06-29 at 12 16 39" src="https://user-images.githubusercontent.com/34686298/176424055-422e1580-99b7-4780-acbf-df053965c5d9.png">


Looking at the batch file in s3 from now. The shape is 
torch.Size([32, 7, 128]). Note 7*128 =896. So need to look at nowcast_dataset code, and see how it handles when there is no data. 
Looks like essentially an empty dataframe is returned - https://github.com/openclimatefix/nowcasting_dataset/blob/main/nowcasting_dataset/data_sources/pv/live.py#L129

Should be able to add some NaNs in here
I like the idea of adding 0, if elevation <10 for provider pvoutput.org. 
I think is sensible to use our knowledge that there is no pv power here, in order to help our models run

Fix patch:
diff --git a/requirements.txt b/requirements.txt
index 738ccd96..1746b377 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -3,7 +3,7 @@ sqlalchemy
 psycopg2-binary
 numpy
 click
-nowcasting_dataset==3.5.14
+nowcasting_dataset==3.6.1
 nowcasting_datamodel==1.1.8
 nowcasting_dataloader==1.8.38
 fsspec


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index fd5d5cf6..2595989c 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -18,7 +18,7 @@
 )
 from nowcasting_datamodel.models.base import Base_Forecast, Base_PV
 from nowcasting_dataset.config.model import Configuration
-from nowcasting_dataset.data_sources.fake.batch import make_random_image_coords_osgb
+from nowcasting_dataset.data_sources.fake.batch import make_image_coords_osgb
 from nowcasting_dataset.dataset.batch import Batch
 
 from nowcasting_forecast import N_GSP
@@ -127,7 +127,7 @@ def nwp_data():
     image_size = 1000
     time_steps = 10
 
-    x, y = make_random_image_coords_osgb(
+    x, y = make_image_coords_osgb(
         size_x=image_size,
         size_y=image_size,
         x_center_osgb=x_center_osgb,