Repository: electricitymaps/electricitymaps-contrib

Issue:
AUS-TAS-KI reports negative wind generation sometimes
Small values, so probably self-consumption. I'll prepare a fix

[Kibana](https://kibana.electricitymap.org/app/kibana#/discover/10af54f0-0c4a-11e9-85c1-1d63df8c862c?_g=(refreshInterval:('$$hashKey':'object:232',display:'5%20minutes',pause:!f,section:2,value:300000),time:(from:now-24h,mode:quick,to:now))&_a=(columns:!(message,extra.key,level),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'96f67170-0c49-11e9-85c1-1d63df8c862c',key:level,negate:!f,params:(query:ERROR,type:phrase),type:phrase,value:ERROR),query:(match:(level:(query:ERROR,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'96f67170-0c49-11e9-85c1-1d63df8c862c',key:extra.key,negate:!f,params:(query:AUS-TAS-KI,type:phrase),type:phrase,value:AUS-TAS-KI),query:(match:(extra.key:(query:AUS-TAS-KI,type:phrase))))),index:'96f67170-0c49-11e9-85c1-1d63df8c862c',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc))):

>November 17th 2020, 17:23:16.743
invalid point: {'zoneKey': 'AUS-TAS-KI', 'datetime': datetime.datetime(2020, 11, 18, 3, 23, 16, 425060, tzinfo=tzfile('/usr/share/zoneinfo/Australia/Currie')), 'production': {'battery discharge': 0, 'biomass': 0.0, 'coal': 0, 'gas': 0, 'hydro': 0, 'nuclear': 0, 'oil': 1.338, 'solar': 0.0, 'wind': -0.003, 'geothermal': 0, 'unknown': 0}, 'storage': {'battery': 0.004}, 'source': 'https://data.ajenti.com.au/KIREIP/index.html', 'schemaVersion': 3}, reason:AUS-TAS-KI: key wind has negative value -0.003
AUS-TAS-KI
ERROR
>November 17th 2020, 17:02:28.794invalid point: {'zoneKey': 'AUS-TAS-KI', 'datetime': datetime.datetime(2020, 11, 18, 3, 2, 27, 648934, tzinfo=tzfile('/usr/share/zoneinfo/Australia/Currie')), 'production': {'battery discharge': 0, 'biomass': 0.0, 'coal': 0, 'gas': 0, 'hydro': 0, 'nuclear': 0, 'oil': 1.217, 'solar': 0.0, 'wind': -0.006, 'geothermal': 0, 'unknown': 0}, 'storage': {'battery': 0.004}, 'source': 'https://data.ajenti.com.au/KIREIP/index.html', 'schemaVersion': 3}, reason:AUS-TAS-KI: key wind has negative value -0.006
AUS-TAS-KI
ERROR
>November 17th 2020, 16:49:52.790invalid point: {'zoneKey': 'AUS-TAS-KI', 'datetime': datetime.datetime(2020, 11, 18, 2, 49, 49, 351083, tzinfo=tzfile('/usr/share/zoneinfo/Australia/Currie')), 'production': {'battery discharge': 0, 'biomass': 0.0, 'coal': 0, 'gas': 0, 'hydro': 0, 'nuclear': 0, 'oil': 1.264, 'solar': 0.0, 'wind': -0.017, 'geothermal': 0, 'unknown': 0}, 'storage': {'battery': 0.004}, 'source': 'https://data.ajenti.com.au/KIREIP/index.html', 'schemaVersion': 3}, reason:AUS-TAS-KI: key wind has negative value -0.017
AUS-TAS-KI
ERROR
AUS-TAS-KI reports negative wind generation sometimes
Small values, so probably self-consumption. I'll prepare a fix

[Kibana](https://kibana.electricitymap.org/app/kibana#/discover/10af54f0-0c4a-11e9-85c1-1d63df8c862c?_g=(refreshInterval:('$$hashKey':'object:232',display:'5%20minutes',pause:!f,section:2,value:300000),time:(from:now-24h,mode:quick,to:now))&_a=(columns:!(message,extra.key,level),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'96f67170-0c49-11e9-85c1-1d63df8c862c',key:level,negate:!f,params:(query:ERROR,type:phrase),type:phrase,value:ERROR),query:(match:(level:(query:ERROR,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'96f67170-0c49-11e9-85c1-1d63df8c862c',key:extra.key,negate:!f,params:(query:AUS-TAS-KI,type:phrase),type:phrase,value:AUS-TAS-KI),query:(match:(extra.key:(query:AUS-TAS-KI,type:phrase))))),index:'96f67170-0c49-11e9-85c1-1d63df8c862c',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc))):

>November 17th 2020, 17:23:16.743
invalid point: {'zoneKey': 'AUS-TAS-KI', 'datetime': datetime.datetime(2020, 11, 18, 3, 23, 16, 425060, tzinfo=tzfile('/usr/share/zoneinfo/Australia/Currie')), 'production': {'battery discharge': 0, 'biomass': 0.0, 'coal': 0, 'gas': 0, 'hydro': 0, 'nuclear': 0, 'oil': 1.338, 'solar': 0.0, 'wind': -0.003, 'geothermal': 0, 'unknown': 0}, 'storage': {'battery': 0.004}, 'source': 'https://data.ajenti.com.au/KIREIP/index.html', 'schemaVersion': 3}, reason:AUS-TAS-KI: key wind has negative value -0.003
AUS-TAS-KI
ERROR
>November 17th 2020, 17:02:28.794invalid point: {'zoneKey': 'AUS-TAS-KI', 'datetime': datetime.datetime(2020, 11, 18, 3, 2, 27, 648934, tzinfo=tzfile('/usr/share/zoneinfo/Australia/Currie')), 'production': {'battery discharge': 0, 'biomass': 0.0, 'coal': 0, 'gas': 0, 'hydro': 0, 'nuclear': 0, 'oil': 1.217, 'solar': 0.0, 'wind': -0.006, 'geothermal': 0, 'unknown': 0}, 'storage': {'battery': 0.004}, 'source': 'https://data.ajenti.com.au/KIREIP/index.html', 'schemaVersion': 3}, reason:AUS-TAS-KI: key wind has negative value -0.006
AUS-TAS-KI
ERROR
>November 17th 2020, 16:49:52.790invalid point: {'zoneKey': 'AUS-TAS-KI', 'datetime': datetime.datetime(2020, 11, 18, 2, 49, 49, 351083, tzinfo=tzfile('/usr/share/zoneinfo/Australia/Currie')), 'production': {'battery discharge': 0, 'biomass': 0.0, 'coal': 0, 'gas': 0, 'hydro': 0, 'nuclear': 0, 'oil': 1.264, 'solar': 0.0, 'wind': -0.017, 'geothermal': 0, 'unknown': 0}, 'storage': {'battery': 0.004}, 'source': 'https://data.ajenti.com.au/KIREIP/index.html', 'schemaVersion': 3}, reason:AUS-TAS-KI: key wind has negative value -0.017
AUS-TAS-KI
ERROR


Discussion:

Resolving patch:
diff --git a/parsers/AUS_TAS_KI.py b/parsers/AUS_TAS_KI.py
--- a/parsers/AUS_TAS_KI.py
+++ b/parsers/AUS_TAS_KI.py
@@ -34,7 +34,13 @@ def get_value(self, hub, method):
         
 def parse_payload(logger, payload):
     technologies_parsed = {}
-    logger.debug(f"serie : {json.dumps(payload)}")
+    if not 'technologies' in payload:
+      raise KeyError(
+        f"No 'technologies' in payload\n"
+        f"serie : {json.dumps(payload)}"
+      )
+    else:
+      logger.debug(f"serie : {json.dumps(payload)}")
     for technology in payload['technologies']:
         assert technology['unit'] == 'kW'
         # The upstream API gives us kW, we need MW
@@ -73,7 +79,7 @@ def fetch_production(zone_key='AUS-TAS-KI', session=None, target_datetime=None,
           'nuclear': 0,
           'oil': technologies_parsed['diesel']*(100-biodiesel_percent)/100,
           'solar': technologies_parsed['solar'],
-          'wind': technologies_parsed['wind'],
+          'wind': 0 if technologies_parsed['wind'] < 0 and technologies_parsed['wind'] > -0.1 else technologies_parsed['wind'], #If wind between 0 and -0.1 set to 0 to ignore self-consumption
           'geothermal': 0,
           'unknown': 0
       },


Test patch: