Repository: ioos/compliance-checker

Issue:
Installation problems on windows
I'm trying to test compliance-checker under windows with anaconda. `pip install compliance-checker` is freaking out with: 

```
HDF5_DIR environment variable not set, checking some standard locations ..
checking C:\Users\christopher.rae ...
checking /usr/local ...
checking /sw ...
checking /opt ...
checking /opt/local ...
checking /usr ...
Traceback (most recent call last):
  File "<string>", line 17, in <module>
  File  c:\users\christopher.rae\appdata\local\temp\1\pip_build_Christopher.Rae\netCDF4\setup.py", line 208, in <module>
    raise ValueError('did not find HDF5 headers')
ValueError: did not find HDF5 headers
```

@dpsnowden suggests it's probably related to the way PyTables or netcdf4-python were installed in conda for windows, and that @rsignell-usgs may have insight having installed related packages on windows. 



Discussion:
Okay, so that particular one is resolved by going to https://code.google.com/p/netcdf4-python/ and doing a manual download and install of netCDF4. Of course, a little further on `pip install compliance-checker` breaks again, this time unable to find swig.exe. More about that elsewhere/later. 

You are going to get stuck on UDUNITS.   The UDUNITS library from Unidata does not build on Windows. 

You're saying we won't be able to run compliance-checker on windows? 

Yes, that's what I'm saying.

At least, until we get Unidata to act on this PR 
https://github.com/Unidata/UDUNITS-2/pull/4
and get it working for MS Visual Studio 2008, so that it will play nice with python 2.7
If you wanted to comment on that PR that it would be useful for you, that would be great!  Maybe it would help Unidata to know that people other than me care....

Kind of a stupid question: is it possible to link some UDUNITS instance that runs remotely, perhaps on some Web server?..

So udunits as a service?   That would be cool, actually!

Pretty interesting solution, @abirger - we could write a small (I decline to say "simple") web service on top of udunits functionality, and hit that remotely - also make compliance-checker only optionally depend on udunits.

I've just filed #62 as a way to keep track of what to do about udunits.

Perhaps this exists in part already. http://coastwatch.pfeg.noaa.gov/erddap/convert/units.html

Has there been any development on getting this to work on windows, or any system. I've done a vanilla install of Anaconda on both a windows system and a linux system and can't seem to get it to work. After the install of Anaconda I've done a fresh install of this package using 'pip install compliance-checker' and received issues with the 'swig.exe' package. When trying to use the example provided here, I get errors that no 'udunitspy' module exists. This problem occurred on both Windows (8.1 Pro) and Ubuntu (12.04 LTS). Below is my testing code for running the checker:

from compliance_checker.runner import ComplianceCheckerCheckSuite

cs = ComplianceCheckerCheckSuite()
dataset = cs.load_dataset("http://data.nodc.noaa.gov/thredds/dodsC/testdata/mbiddle/45029_sea_water_temperature.nc")
groups = cs.run(dataset, 'acdd')
scores = groups['acdd']

I just wanted to get my testing attempts out in the open so everyone knows the trouble we've been having with installing this package.

@mbiddle-nodc I do not believe that Anaconda has UDUnits installed.  Try to install with pip:

```
pip install udunitspy
```

Failure to install swig.exe and the lack of udunitspy on windows was where I essentially gave up on this problem. At least as far as I am concerned, I do not have access to a sanctioned linux machine yet, so all my attempts at investigating further on this have been abandoned for now. 

I could not get `pip install udunitspy` to work. 

Thanks for the quick response. I should have noted, that when I tried to install udunitspy on it's own (for both systems), it attempts to install but the last line it outputs is:
error: command 'swig.exe' failed: No such file or directory.

So, I'm assuming this is an issue with the udunitspy package installation? I'm not all that fluent in python as of yet, so simple explanations are appreciated.

udunitspy depends heavily on libudunits and it also depends on libudunits being installed on a Unix-based system in a familiar directory structure. 
https://github.com/blazetopher/udunitspy/blob/master/setup.py#L51-L55

In summary, udunitspy isn't supported on windows. 

There remains still the reported problems installing on Linux systems.
@mbiddle-nodc should probably start a new issue about installation problems on linux without the distraction of windows in the subject line.

On Mon, Aug 18, 2014 at 10:28 AM, Luke Campbell notifications@github.com
wrote:

> udunitspy depends heavily on libudunits and it also depends on libudunits
> being installed on a Unix-based system in a familiar directory structure.
> https://github.com/blazetopher/udunitspy/blob/master/setup.py#L51-L55
> 
> In summary, udunitspy isn't supported on windows.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ioos/compliance-checker/issues/61#issuecomment-52499766
> .

## 

## 

## 

# 

Dr. Christopher M. Duncombe Rae       c deirdre.byrne@noaa.gov
hristopher.duncombe.rae@noaa.gov
Physical/Data Scientist, IOOS/NOAA, Suite 1225,
1100 Wayne Avenue, Silver Spring, MD 20910, USA
Tel: +1-301-427-2450     Fax:  +1-301-427-2073

@mbiddle-nodc maybe I can touch base with you, I have time today and I can look at your issue. I don't think I can address the windows issues but I can certainly get the linux issues fixed. 

@lukecampbell @mbiddle-nodc Please post the resolution to any issues here after you've working things out.

Thanks!

That sounds good. I'm upgrading my Ubuntu to 14.04 to see if that makes any difference. 

I'm going to create a new issue for this, since it's not windows related.
https://github.com/ioos/compliance-checker/issues/65

As for the windows installation, this will be an outstanding issue until Unidata releases a udunits library that compiles on windows. 

@lukecampbell , actually we've been working with Unidata on this, with the goal of getting udunits going as a conda module for windows users, ad udunits on windows is a blocker not only for the compliance checker on windows, but for Iris on windows.  They just got it going 4 days ago
https://github.com/Unidata/UDUNITS-2/pull/17
and I tried building it on Windows this morning and it worked:
https://github.com/Unidata/UDUNITS-2/issues/18

A small bird told me that windows builds for udunits are available via conda: https://binstar.org/rsignell/udunits2/files

Yes, but before the compliance-checker will work on windows, either the dependency on `udunitspy` will need to be replaced with something else, or else the build script for `udunitspy` will have to be modified to work on windows.  I'm not sure @blazetopher anticipated that use case.

```
c:> conda build udunitspy
...
running install
running bdist_egg
running build_ext
building '_udunits2_c' extension
swigging udunitspy/udunits2_c.i to udunitspy/udunits2_c_wrap.cpp
c:\programs\swigwin-3.0.2\swig.exe -python -c++ -I/usr/include -I/usr/local/incl
ude -I/usr/include/udunits2 -o udunitspy/udunits2_c_wrap.cpp udunitspy/udunits2_
c.i
udunitspy\udunits2_c.i(90) : Error: Unable to find 'udunits2.h'
udunitspy\udunits2_c.i(116) : Error: Unable to find 'converter.h'
error: command 'c:\\programs\\swigwin-3.0.2\\swig.exe' failed with exit status 1


c:\programs\Anaconda\conda-bld\work>if errorlevel 1 exit 1
Command failed: C:\Windows\system32\cmd.exe /c bld.bat

```

@rsignell-usgs , @daf , @lukecampbell : Any news from the Windows battle-front? Should I invest time into squeezing my way through the countless problems with Windows native installation, or rather spend it on Linux virtual machine?..

@abirger - I would continue to stick with Linux at this point.

The one hurdle for running the compliance checker on Windows is the dependence on  [udunitspy](https://github.com/blazetopher/udunitspy).    Although the udunits2 library is available on Windows, the udunitpy build script uses unix-like syntax that doesn't work on windows.   See this issue: https://github.com/Unidata/UDUNITS-2/issues/3#issuecomment-72639479

@kknee, I bet if we had 30 min of @blazetopher time we could fix this.  Let me know if that's posssible. 

@rsignell-usgs,  @blazetopher chatted about this a while back and my impression was that it would take more than 30 min.  

@blazetopher - please chime in if I misunderstood.  

@blazetopher tells me @lukecampbell is our man.

@rsignell-usgs , will it help if I have MinGW/msys installed, and can run Unix command in Bash scripts?..

@abirger, we could hack it that way, but Cmake would be a better route for the future.  For someone who knows Cmake, I think it would take 30 min.   For someone who doesn't, substantially more time, as @kknee noted.

@lukecampbell says that udunitspy right now is not compatible with Windows. Getting udunitspy to be compatible with windows isn't in the scope of compliance checker right now. Cmake isn't the issue I think. udunitspy uses swig to create the bindings compile either a (.dll, .so or .dylib) which allows the python interpreter to call udunits methods. I don't know what the level of effort is to update our setup.py and/or swig interface definitions (since DLL's have dramatically different binding types than .so or .dylib). 

I would actually be considerably more in favor of removing udunitspy from compliance checker and replacing it with an in-line units module. 

(pssst: #62)

Yeah, that's from our recommendations early on. 

Sure, it's Swig.  But basically isn't it just taking the lib and include requirements from here
https://github.com/blazetopher/udunitspy/blob/master/setup.py#L34-L70
and putting it into the CMake config file?
http://www.swig.org/Doc1.3/Introduction.html#Introduction_build_system

So when I said 30 min, I meant 30 min for someone who knew Swig working with someone who knew CMake (could be the same person, could be two people).    I was kind of interested in this conversion process for reasons beyond just the compliance-checker.

But I realize that to just get the job done here for this application, the web service would be an easy route that would require no knowledge of Cmake or Swig.  :grin:

Here's an example of a conda-recipe that uses Cmake for Mac, Linux and Windows:
https://github.com/conda/conda-recipes/tree/master/libdynd

@abirger Will document steps required to use VirtualBox and update documentation accordingly!


Resolving patch:
diff --git a/compliance_checker/acdd.py b/compliance_checker/acdd.py
--- a/compliance_checker/acdd.py
+++ b/compliance_checker/acdd.py
@@ -4,7 +4,7 @@
 from pkgutil import get_data
 
 from dateutil.parser import parse as parse_dt
-from udunitspy import Unit
+from cf_units import Unit
 
 from compliance_checker.base import BaseCheck, BaseNCCheck, check_has, score_group, Result
 from compliance_checker.cf.cf import _possiblexunits, _possibleyunits
@@ -15,7 +15,7 @@ class ACDDBaseCheck(BaseCheck):
     ###############################################################################
     #
     # HIGHLY RECOMMENDED
-    # 
+    #
     ###############################################################################
 
     @check_has(BaseCheck.HIGH)
diff --git a/compliance_checker/cf/util.py b/compliance_checker/cf/util.py
--- a/compliance_checker/cf/util.py
+++ b/compliance_checker/cf/util.py
@@ -3,7 +3,7 @@
 import itertools
 from collections import defaultdict
 from lxml import etree
-from udunitspy import Unit, UdunitsError, Converter
+from cf_units import Unit
 from netCDF4 import Dimension, Variable
 from pkgutil import get_data
 
@@ -24,7 +24,7 @@
            "depths", "DEPTHS",
            "height", "HEIGHT",
            "altitude", "ALTITUDE",
-           "alt", "ALT", 
+           "alt", "ALT",
            "Alt", "Altitude",
            "h", "H",
            "s_rho", "S_RHO",
@@ -81,25 +81,25 @@
                 'degreeN'
                     ]
 
-_possibletunits = ['day', 
-                'days', 
-                'd', 
-                'hour', 
-                'hours', 
-                'hr', 
-                'hrs', 
-                'h', 
-                'year', 
-                'years', 
-                'minute', 
-                'minutes', 
-                'm', 
-                'min', 
-                'mins', 
-                'second', 
-                'seconds', 
-                's', 
-                'sec', 
+_possibletunits = ['day',
+                'days',
+                'd',
+                'hour',
+                'hours',
+                'hr',
+                'hrs',
+                'h',
+                'year',
+                'years',
+                'minute',
+                'minutes',
+                'm',
+                'min',
+                'mins',
+                'second',
+                'seconds',
+                's',
+                'sec',
                 'secs'
                 ]
 
@@ -250,31 +250,25 @@ def __iter__(self):
 
 def units_known(units):
     try:
-        Unit(str(units))
-    except UdunitsError:
+        Unit(units)
+    except ValueError:
         return False
     return True
 
+
 def units_convertible(units1, units2, reftimeistime=True):
     """Return True if a Unit representing the string units1 can be converted
     to a Unit representing the string units2, else False."""
     try:
-        Converter(str(units1), str(units2))
-    except UdunitsError:
+        u1 = Unit(units1)
+        u2 = Unit(units2)
+    except ValueError:
         return False
-
-    u1 = Unit(str(units1))
-    u2 = Unit(str(units2))
-    return u1.are_convertible(u2)
+    return u1.is_convertible(units2)
 
 def units_temporal(units):
-    r = False
-    try:
-        u = Unit('seconds since 1900-01-01')
-        r = u.are_convertible(str(units))
-    except UdunitsError:
-        return False
-    return r
+    u = Unit(units)
+    return u.is_time_reference()
 
 def map_axes(dim_vars, reverse_map=False):
     """
@@ -325,14 +319,14 @@ def is_time_variable(varname, var):
 def is_vertical_coordinate(var_name, var):
     """
     Determines if a variable is a vertical coordinate variable
-    
+
     4.3
     A vertical coordinate will be identifiable by: units of pressure; or the presence of the positive attribute with a
     value of up or down (case insensitive).  Optionally, the vertical type may be indicated additionally by providing
     the standard_name attribute with an appropriate value, and/or the axis attribute with the value Z.
     """
     # Known name
-    satisfied = var_name.lower() in _possiblez 
+    satisfied = var_name.lower() in _possiblez
     satisfied |= getattr(var, 'standard_name', '') in _possiblez
     # Is the axis set to Z?
     satisfied |= getattr(var, 'axis', '').lower() == 'z'


Test patch: