Repository: ioos/compliance-checker

Issue:
checker appears to adjust timezone incorrectly
I'm running cchecker.py version 3.0.0 on a local netcdf file.  The times in the metadata are all of the form:
	time:units = "minutes since 2008-01-01T00:00:00-10:00" ;
		:time_coverage_start = "2016-08-28T01:15:00-10:00" ;
		:time_coverage_end = "2016-08-28T23:00:00-10:00" ;
The actual time values in the file are:
time = 4553355, 4553370, 4553385, 4553400, 4553415, 4553430, 4553445, 
    4553460, 4553475, 4553490, 4553505, 4553520, 4553535, 4553550, 4553565, 
    4553580, 4553715, 4553730, 4553745, 4553760, 4553775, 4553790, 4553805, 
    4553820, 4553835, 4553850, 4553865, 4553880, 4553895, 4553910, 4553925, 
    4553940, 4553955, 4553970, 4553985, 4554000, 4554015, 4554030, 4554045, 
    4554060, 4554075, 4554090, 4554105, 4554120, 4554135, 4554150, 4554165, 
    4554180, 4554195, 4554210, 4554225, 4554240, 4554255, 4554270, 4554285, 
    4554300, 4554315, 4554330, 4554345, 4554360, 4554375, 4554390, 4554405, 
    4554420, 4554435, 4554450, 4554465, 4554480, 4554495, 4554510, 4554525, 
    4554540, 4554555, 4554570, 4554585, 4554600, 4554615, 4554630, 4554645, 
    4554660 ;

i believe these to be consistent with the time_coverage_start and time_coverage_end.  However, the compliance checker returns the following warning:

Name                             Priority:     Score:Reasoning
--------------------------------------------------------------------------------
time_coverage_extents_match            :2:     0/ 2 : Date time mismatch between
                                                      time_coverage_start and
                                                      actual time values
                                                      2016-08-28T11:15:00
                                                      (time_coverage_start) !=
                                                      2016-08-27T15:15:00
                                                      (time[0]), Date time
                                                      mismatch between
                                                      time_coverage_end and
                                                      actual time values
                                                      2016-08-29T09:00:00
                                                      (time_coverage_end) !=
                                                      2016-08-28T13:00:00
                                                      (time[N])

It would appear that the compliance checker is attempting to convert from UTC to local time incorrectly, perhaps an addition/subtraction swap (?).  


Discussion:
Could you include a link to the netCDF file please?
Hi Luke,

We don't have these files on the server yet, but I've attached an 
example.  The python checker cchecker.py version 3.0.0 gives the 
following (see bottom part):

Running Compliance Checker on the dataset from: ../wqb04_2016_12_15.nc


--------------------------------------------------------------------------------
                      The dataset scored 68 out of 70 points
                              during the acdd check
--------------------------------------------------------------------------------
                                Scoring Breakdown:


                                  High Priority
--------------------------------------------------------------------------------
     Name                            :Priority: Score
Conventions                             :3:     2/2
keywords                                :3:     1/1
summary                                 :3:     1/1
title                                   :3:     1/1
varattr                                 :3:    51/51


                                 Medium Priority
--------------------------------------------------------------------------------
     Name                            :Priority: Score
acknowledgment/acknowledgement          :2:     1/1
date_created_is_iso                     :2:     1/1
date_issued_is_iso                      :2:     1/1
date_metadata_modified_is_iso           :2:     1/1
date_modified_is_iso                    :2:     1/1
geospatial_lat_extents_match            :2:     2/2
geospatial_lon_extents_match            :2:     2/2
geospatial_vertical_extents_match       :2:     2/2
no_blanks_in_id                         :2:     1/1
time_coverage_extents_match             :2:     0/2


--------------------------------------------------------------------------------
                   Reasoning for the failed tests given below:


Name                             Priority:     Score:Reasoning
--------------------------------------------------------------------------------
time_coverage_extents_match            :2:     0/ 2 : Date time mismatch 
between
time_coverage_start and
                                                       actual time values
2016-12-15T10:00:00
(time_coverage_start) !=
2016-12-14T14:00:00
                                                       (time[0]), Date time
                                                       mismatch between
time_coverage_end and
                                                       actual time values
2016-12-16T09:45:00
(time_coverage_end) !=
2016-12-15T13:45:00
                                                       (time[N])
WARNING: The following exceptions occured during the acdd checker 
(possibly indicate compliance checker issues):
acdd.check_recommended: invalid literal for float(): -155.082,


On 12/29/16 10:24 AM, Luke Campbell wrote:
>
> Could you include a link to the netCDF file please?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub 
> <https://github.com/ioos/compliance-checker/issues/455#issuecomment-269688198>, 
> or mute the thread 
> <https://github.com/notifications/unsubscribe-auth/AEkrlNi6GLWoJJrnIGFxZ-ngcBfNavSLks5rNBcWgaJpZM4LUWuh>.
>


@jpotemra the attachment didn't come through in github's issue page.  

@lukecampbell there's a link to a sample file in Jim's ioos_tech post, I think https://groups.google.com/d/msg/ioos_tech/-FkfB_OAWRI/HgJV5GuxCQAJ is a link to his message but who knows how to actually operate google groups...
This appears to be due the use of `netCDF4.num2date` here: https://github.com/ioos/compliance-checker/blob/d17a9c54a2c37e85959a11562c9acc4e2753780b/compliance_checker/acdd.py#L474-L484

Documentation for `num2date` says (emphasis added by me):

> Return datetime objects given numeric time values. The units of the numeric time values are described by the units argument and the calendar keyword. *The returned datetime objects represent UTC with no time-zone offset, even if the specified units contain a time-zone offset.*
Appears related to https://github.com/Unidata/netcdf4-python/issues/685.  Upgrading the `netCDF4` version to 1.3.0 should fix this.

Resolving patch:
diff --git a/compliance_checker/acdd.py b/compliance_checker/acdd.py
--- a/compliance_checker/acdd.py
+++ b/compliance_checker/acdd.py
@@ -16,6 +16,7 @@
 from compliance_checker.cf.util import _possiblexunits, _possibleyunits
 from compliance_checker.util import datetime_is_iso, dateparse
 from compliance_checker import cfutil
+import pendulum
 from pygeoif import from_wkt
 
 
@@ -472,8 +473,15 @@ def check_time_extents(self, ds):
 
         # Time should be monotonically increasing, so we make that assumption here so we don't have to download THE ENTIRE ARRAY
         try:
-            time0 = num2date(ds.variables[timevar][0], ds.variables[timevar].units)
-            time1 = num2date(ds.variables[timevar][-1], ds.variables[timevar].units)
+            # num2date returns as naive date, but with time adjusted to UTC
+            # we need to attach timezone information here, or the date
+            # subtraction from t_min/t_max will assume that a naive timestamp is
+            # in the same time zone and cause erroneous results.
+            # Pendulum uses UTC by default, but we are being explicit here
+            time0 = pendulum.instance(num2date(ds.variables[timevar][0],
+                                      ds.variables[timevar].units), 'UTC')
+            time1 = pendulum.instance(num2date(ds.variables[timevar][-1],
+                                      ds.variables[timevar].units), 'UTC')
         except:
             return Result(BaseCheck.MEDIUM,
                           False,


Test patch:
diff --git a/compliance_checker/tests/test_acdd.py b/compliance_checker/tests/test_acdd.py
--- a/compliance_checker/tests/test_acdd.py
+++ b/compliance_checker/tests/test_acdd.py
@@ -435,7 +435,11 @@ def test_time_extents(self):
         time_var[:] = [1451692800]  # 20160102T000000Z in seconds since epoch
         empty_ds.time_coverage_start = '20160102T000000Z'
         empty_ds.time_coverage_end = '20160102T000000Z'
-
-
+        result = self.acdd.check_time_extents(empty_ds)
+        self.assert_result_is_good(result)
+        # try the same thing with time offsets
+        time_var.units = 'seconds since 1970-01-01 00:00:00-10:00'
+        empty_ds.time_coverage_start = '20160102T000000-1000'
+        empty_ds.time_coverage_end = '20160102T000000-1000'
         result = self.acdd.check_time_extents(empty_ds)
         self.assert_result_is_good(result)