Repository: electricitymaps/electricitymaps-contrib

Issue:
Italy South and Sicily need parser update
Production for Italian micro-bidding zones Priolo, Foggia and Brindisi isn't reported to ENTSO-E anymore since 1st January 2019. Probably it now is included in the respective surrounding zones? This is why we miss Italy South and Sicily on the map. Only micro-zone Rossano (included in our "Italy South") still has data which still must be added to ENTSO-E's "South".


Discussion:
I see this error for Italy South quite often;

```shell
Traceback (most recent call last):
  File "/home/feeder/lib/fetch_data.py", line 181, in launch_parsers
    objs = res.get(timeout)
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/feeder/lib/fetch_data.py", line 179, in <lambda>
    **parser_kwargs))
  File "/home/contrib/parsers/ENTSOE.py", line 947, in fetch_production_aggregate
    zone_key)
  File "/home/contrib/parsers/ENTSOE.py", line 911, in merge_production_outputs
    merge_source = parser_outputs[0][0]['source']
TypeError: 'NoneType' object is not subscriptable
```

and this one for Sicily;

```shell
Traceback (most recent call last):
  File "/home/feeder/lib/fetch_data.py", line 181, in launch_parsers
    objs = res.get(timeout)
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/feeder/lib/fetch_data.py", line 179, in <lambda>
    **parser_kwargs))
  File "/home/contrib/parsers/ENTSOE.py", line 947, in fetch_production_aggregate
    zone_key)
  File "/home/contrib/parsers/ENTSOE.py", line 914, in merge_production_outputs
    for output in parser_outputs
  File "/home/contrib/parsers/ENTSOE.py", line 914, in <listcomp>
    for output in parser_outputs
  File "/usr/local/lib/python3.6/site-packages/pandas/core/frame.py", line 3146, in set_index
    level = frame[col]._values
  File "/usr/local/lib/python3.6/site-packages/pandas/core/frame.py", line 2139, in __getitem__
    return self._getitem_column(key)
  File "/usr/local/lib/python3.6/site-packages/pandas/core/frame.py", line 2146, in _getitem_column
    return self._get_item_cache(key)
  File "/usr/local/lib/python3.6/site-packages/pandas/core/generic.py", line 1842, in _get_item_cache
    values = self._data.get(item)
  File "/usr/local/lib/python3.6/site-packages/pandas/core/internals.py", line 3843, in get
    loc = self.items.get_loc(item)
  File "/usr/local/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2527, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 117, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 139, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1265, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1273, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'datetime'
```
ENTSOE looks to have data present at the moment for both zones, what do we need to change?
@systemcatch right, we have data for both zones, but we add some very tiny bidding zones on top. These don't seem to exist ( / don't seem to report data) anymore since 1st January, so we need to exclude them from the sum of the surrounding zones. I think the errors come from calculating production of big zone + "N/A" of small zone.

Resolving patch:
diff --git a/parsers/ENTSOE.py b/parsers/ENTSOE.py
--- a/parsers/ENTSOE.py
+++ b/parsers/ENTSOE.py
@@ -847,8 +847,7 @@ def fetch_production(zone_key, session=None, target_datetime=None,
 
 
 ZONE_KEY_AGGREGATES = {
-    'IT-SIC': ['IT-SIC', 'IT-PR'],
-    'IT-SO': ['IT-FO', 'IT-BR', 'IT-RO', 'IT-SO'],
+    'IT-SO': ['IT-RO', 'IT-SO'],
 }
 
 


Test patch: