Repository: electricitymaps/electricitymaps-contrib

Issue:
Australia AEMO parser sometimes returns double the production 
I'm guessing at some points, the source CSV contains information about production at two different timestamps (e.g. 03:30 as well as 03:25). The carbon intensity remains mostly correct, except the effect of solar production (which comes from a different source) is halved. Still, would be good to only return data for one timestamp per plant.

Compare the following screenshots, where generation from coal doubles from 5.32 GB to 10.6 GW - and so does the reported installed coal capacity.

<img width="315" alt="au bug 1" src="https://user-images.githubusercontent.com/47415/29496801-4cbf7698-85da-11e7-9d0f-2294e66d5ba9.PNG">
<img width="332" alt="au bug 2" src="https://user-images.githubusercontent.com/47415/29496802-4fd8db9e-85da-11e7-83a9-a30cdb26adbb.PNG">



Discussion:
There could be a check at the start where the max or most common timestamp is found.  Then any key that has a different timestamp to that is dropped.

Resolving patch:
diff --git a/parsers/AU.py b/parsers/AU.py
--- a/parsers/AU.py
+++ b/parsers/AU.py
@@ -340,6 +340,13 @@ def fetch_production(country_code=None, session=None):
         'source': 'aremi.nationalmap.gov.au, pv-map.apvi.org.au',
     }
 
+    # It's possible that the csv sometimes contains several timestamps.
+    # https://github.com/tmrowco/electricitymap/issues/704
+    # Find the latest timestamp.
+    timestamps = df['Most Recent Output Time (AEST)'].dropna().values
+    latest_ts = max([arrow.get(x) for x in timestamps.tolist()])
+    data['datetime'] = latest_ts.datetime
+
     for rowIndex, row in df.iterrows():
         station = row['Station Name']
         fuelsource = row['Fuel Source - Descriptor']
@@ -384,6 +391,18 @@ def fetch_production(country_code=None, session=None):
             print(row)
             continue
 
+        # Parse the datetime and check it matches the most recent one.
+        try:
+            plant_timestamp = arrow.get(row['Most Recent Output Time (AEST)']).datetime
+        except (OSError, ValueError):
+            # ignore invalid dates, they might be parsed as NaN
+            continue
+        else:
+            # if plant_timestamp could be parsed successfully,
+            # check plant_timestamp equals latest_timestamp and drop plant otherwise
+            if plant_timestamp != data['datetime']:
+                continue
+
         # Initialize key in data dictionaries if not set
         if key not in data['production']:
             data['production'][key] = 0.0
@@ -395,21 +414,6 @@ def fetch_production(country_code=None, session=None):
         data['production'][key] = max(data['production'][key], 0)
         data['capacity'][key] = max(data['capacity'][key], 0)
 
-        # Parse the datetime and return a python datetime object
-        try:
-            plant_timestamp = arrow.get(row['Most Recent Output Time (AEST)']).datetime
-            # TODO: We should check it's not too old..
-        except (OSError, ValueError):
-            # ignore invalid dates, they might be parsed as NaN
-            continue
-        else:
-            # if plant_timestamp could be parsed successfully,
-            # set to max of plant production timestamp and current max timestamp in dict (if any)
-            current_datetime = data.get('datetime', None)
-            if current_datetime:
-                data['datetime'] = max(plant_timestamp, current_datetime)
-            else:
-                data['datetime'] = plant_timestamp
 
     # find distributed solar production and add it in
     session = session or requests.session()


Test patch: