Repository: electricitymaps/electricitymaps-contrib

Issue:
ENTSO-E Pumped Hydro consumption missing 
It seems that consumption values for pumped hydro (storage) are missing for countries based on ENTSO-E data. 
They are reported on their website, so I guess something is wrong in the parser. 


Discussion:
Which countries are you looking at? I checked now on the frontend and the UK, France and Germany are all reporting pumped storage values.
Belgium, Germany, Portugal, and more...
The positive values (when producing electricity) are shown, but the negative values (when storing electricity) are not. 
Ah yes I see what you mean. So for example hydro storage for Austria is reported as -.0.0 at 03:00 UTC this morning:

![Screenshot from 2020-11-13 17-55-12](https://user-images.githubusercontent.com/693233/99104883-81c16c00-25d9-11eb-8ab0-b002e266d09e.png)

However ENTSOE [reports](https://transparency.entsoe.eu/generation/r2/actualGenerationPerProductionType/show?name=&defaultValue=false&viewType=GRAPH&areaType=BZN&atch=false&datepicker-day-offset-select-dv-date-from_input=D&dateTime.dateTime=13.11.2020+03:00|UTC|DAYTIMERANGE&dateTime.endDateTime=13.11.2020+04:00|UTC|DAYTIMERANGE&area.values=CTY|10YAT-APG------L!BZN|10YAT-APG------L&productionType.values=B01&productionType.values=B02&productionType.values=B03&productionType.values=B04&productionType.values=B05&productionType.values=B06&productionType.values=B07&productionType.values=B08&productionType.values=B09&productionType.values=B10&productionType.values=B11&productionType.values=B12&productionType.values=B13&productionType.values=B14&productionType.values=B20&productionType.values=B15&productionType.values=B16&productionType.values=B17&productionType.values=B18&productionType.values=B19&dateTime.timezone=UTC&dateTime.timezone_input=UTC) that at this time hydro storage consumption was -422MW (and there was no hydro storage production at this time).
 
I've found the cause, it's because of [this commit](https://github.com/tmrowco/electricitymap-contrib/commit/97865b8d33fbfe643aeba2fcff53ecb8b727786c?branch=97865b8d33fbfe643aeba2fcff53ecb8b727786c&diff=unified) from a month ago that ignores consumption values when calculating production. If you revert the code that was changed then we get the pumped storage data again. @FelixDQ @PaulCornelissen (authors of the commit) was this effect intended in your commit?
Nope, that was certainly not the intended effect. You can read the history here: #2676 and #2638 

In short: In NL there's sometimes significant self consumption at power plants (e.g. CHP plants at large consumers). Sometimes it's more than the production for an entire generation type (mostly for unknown). That would cause the production for unknown to be negative and the parser would fail. 

I suppose that we should add an extra condition to differentiate between consumption for hydro(storage) and other types of generation.
> Nope, that was certainly not the intended effect. You can read the history here: #2676 and #2638
> 
> In short: In NL there's sometimes significant self consumption at power plants (e.g. CHP plants at large consumers). Sometimes it's more than the production for an entire generation type (mostly for unknown). That would cause the production for unknown to be negative and the parser would fail.
> 
> I suppose that we should add an extra condition to differentiate between consumption for hydro(storage) and other types of generation.

Ok I will see if I can fix it.

Resolving patch:
diff --git a/parsers/ENTSOE.py b/parsers/ENTSOE.py
--- a/parsers/ENTSOE.py
+++ b/parsers/ENTSOE.py
@@ -620,18 +620,22 @@ def parse_production(xml_text):
         is_production = len(timeseries.find_all('inBiddingZone_Domain.mRID'.lower())) > 0
         psr_type = timeseries.find_all('mktpsrtype')[0].find_all('psrtype')[0].contents[0]
 
-        if len(timeseries.find_all('inBiddingZone_Domain.mRID'.lower())) > 0:
-            for entry in timeseries.find_all('point'):
-                quantity = float(entry.find_all('quantity')[0].contents[0])
-                position = int(entry.find_all('position')[0].contents[0])
-                datetime = datetime_from_position(datetime_start, position, resolution)
-                try:
-                    i = datetimes.index(datetime)
+        for entry in timeseries.find_all('point'):
+            quantity = float(entry.find_all('quantity')[0].contents[0])
+            position = int(entry.find_all('position')[0].contents[0])
+            datetime = datetime_from_position(datetime_start, position, resolution)
+            try:
+                i = datetimes.index(datetime)
+                if is_production:
                     productions[i][psr_type] += quantity
-                except ValueError:  # Not in list
-                    datetimes.append(datetime)
-                    productions.append(defaultdict(lambda: 0))
-                    productions[-1][psr_type] = quantity if is_production else -1 * quantity
+                elif psr_type in ENTSOE_PARAMETER_GROUPS['storage']['hydro storage']:
+                    # Only include consumption if it's for hydro storage. In other cases
+                    # it is power plant self-consumption which should be ignored.
+                    productions[i][psr_type] -= quantity
+            except ValueError:  # Not in list
+                datetimes.append(datetime)
+                productions.append(defaultdict(lambda: 0))
+                productions[-1][psr_type] = quantity if is_production else -1 * quantity
     return productions, datetimes
 
 


Test patch: