Repository: electricitymaps/electricitymaps-contrib

Issue:
CL-SIC parser returns wrong datetime
CL-SIC parser returns datetime of 0018 instead of 2018, as illustrated below:

`datetime: 0018-02-18 (06:43:00.000) CET`
`production: {"unknown":0,"coal":2206.6,"hydro":2416.39660,"wind":309.314,....}`
`updatedAt: 2018-02-20 (23:40:06.979) CET`

@systemcatch any idea?



Discussion:
This reminds me we should also update the validation functions to make sure we don't insert something that's < year 2000
I'm fairly sure it's https://github.com/tmrowco/electricitymap/blob/master/parsers/CL_SIC.py#L334 that is causing the problem. But I already see a better way to get the datetime that won't be so convoluted.
Something else I've noticed is that the site returns status code 200 on it's 404 page which messes up the logic to get the right file. Another thing to fix.
Happened again. 
@maxbellec can you add the check?
@systemcatch any clue as to why this is happening?
It returned year < 2000 again?

@corradio it's working now when I run it manually. That error indicates the file was there with it's usual name, but the structure must have been totally different for https://github.com/tmrowco/electricitymap/blob/master/parsers/CL_SIC.py#L262 to fail. Let's keep and eye on it and see if it happens again, they might change the file type near the end of the month or something.
adding the check yes

Resolving patch:
diff --git a/parsers/lib/quality.py b/parsers/lib/quality.py
--- a/parsers/lib/quality.py
+++ b/parsers/lib/quality.py
@@ -8,11 +8,26 @@ class ValidationError(ValueError):
     pass
 
 
+def validate_reasonable_time(item, k):
+    data_time = arrow.get(item['datetime'])
+    if data_time.year < 2000:
+        raise ValidationError("Data from %s can't be before year 2000, it was "
+                              "%s" % (k, data_time))
+
+    arrow_now = arrow.utcnow()
+    if data_time > arrow_now:
+        raise ValidationError(
+            "Data from %s can't be in the future, data was %s, now is "
+            "%s" % (k, data_time, arrow_now))
+
+
+
 def validate_consumption(obj, zone_key):
     # Data quality check
     if obj['consumption'] is not None and obj['consumption'] < 0:
         raise ValidationError('%s: consumption has negative value '
                               '%s' % (zone_key, obj['consumption']))
+    validate_reasonable_time(obj, zone_key)
 
 
 def validate_exchange(item, k):
@@ -24,13 +39,7 @@ def validate_exchange(item, k):
     if type(item['datetime']) != datetime.datetime:
         raise ValidationError('datetime %s is not valid for %s' %
                               (item['datetime'], k))
-    data_time = arrow.get(item['datetime'])
-    if data_time > arrow.now():
-        raise ValidationError("Data from %s can't be in the future, data was "
-                              "%s, now is %s" % (k, data_time, arrow.now()))
-    if data_time.year < 2000:
-        raise ValidationError("Data from %s can't be before year 2000, it was "
-                              "%s" % (k, data_time))
+    validate_reasonable_time(item, k)
 
 
 def validate_production(obj, zone_key):
@@ -48,12 +57,6 @@ def validate_production(obj, zone_key):
     if (obj.get('zoneKey', None) or obj.get('countryCode', None)) != zone_key:
         raise ValidationError("Zone keys %s and %s don't match in %s" %
                               (obj.get('zoneKey', None), zone_key, obj))
-    data_time = arrow.get(obj['datetime'])
-    arrow_now = arrow.utcnow()
-    if data_time > arrow_now:
-        raise ValidationError(
-            "Data from %s can't be in the future, data was %s, now is "
-            "%s" % (zone_key, data_time, arrow_now))
 
     if ((obj.get('production', {}).get('unknown', None) is None and
          obj.get('production', {}).get('coal', None) is None and
@@ -69,3 +72,4 @@ def validate_production(obj, zone_key):
         if v < 0:
             raise ValidationError('%s: key %s has negative value %s' %
                                   (zone_key, k, v))
+    validate_reasonable_time(obj, zone_key)


Test patch: