Repository: ARM-DOE/ACT

Issue:
Xarray not able to read old MMCR data
Some older data might not be compatible with xarray.  sgpmmcrmomC1.b1.20041107 data yielded this error:

xarray.core.variable.MissingDimensionsError: 'heights' has more than 1-dimension and the same name as one of its dimensions ('mode', 'heights'). xarray disallows such variables because they conflict with the coordinates used to label dimensions.

Is there a way around this?


Discussion:
what does a ncdump look like? Sounds like the format is horrible! 
we might need to make a custom ingest from netCDF4 in this case. 

You can specify decode_coords=False to xarray.open_dataset and that should at least disable the dimension labelling for nasty datasets like this.

________________________________
From: Adam Theisen <notifications@github.com>
Sent: Wednesday, May 22, 2019 9:33:25 AM
To: ANL-DIGR/ACT
Cc: Subscribed
Subject: [ANL-DIGR/ACT] Xarray not able to read old MMCR data (#67)


Some older data might not be compatible with xarray. sgpmmcrmomC1.b1.20041107 data yielded this error:

xarray.core.variable.MissingDimensionsError: 'heights' has more than 1-dimension and the same name as one of its dimensions ('mode', 'heights'). xarray disallows such variables because they conflict with the coordinates used to label dimensions.

Is there a way around this?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://github.com/ANL-DIGR/ACT/issues/67?email_source=notifications&email_token=AFIQA5DSOSEQ26RJ2ERS3M3PWVKTLA5CNFSM4HOU6TA2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GVHKM6Q>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFIQA5FHMG6NXEAFKAGDYNLPWVKTLANCNFSM4HOU6TAQ>.

netcdf sgpmmcrmomC1.b1.20041107.000014 {
dimensions:
        time = UNLIMITED ; // (44959 currently)
        mode = 10 ;
        namelength = 64 ;
        heights = 167 ;
        hourly = 24 ;
variables:
        int base_time ;
                base_time:string = "7-Nov-2004,0:00:14 GMT" ;
                base_time:long_name = "Base time in Epoch" ;
                base_time:units = "seconds since 1970-1-1 0:00:00 0:00" ;
        double time_offset(time) ;
                time_offset:long_name = "Time offset from base_time" ;
                time_offset:units = "seconds since 2004-11-07 00:00:14 0:00" ;
        double time(time) ;
                time:long_name = "Time offset from midnight" ;
                time:units = "seconds since 2004-11-07 00:00:00 0:00" ;
 
        float heights(mode, heights) ;
                heights:long_name = "Range Heights (center of radar sample volume)" ;
                heights:units = "m MSL" ;
                heights:missing_value = -9999.f ;
        int DataQualityStatus(time) ;
                DataQualityStatus:long_name = "Data Quality Status" ;
                DataQualityStatus:units = "code" ;
                DataQualityStatus:missing_value = -9999 ;
        float MeanDopplerVelocity(time, heights) ;
                MeanDopplerVelocity:long_name = "Mean Doppler Velocity" ;
                MeanDopplerVelocity:units = "m/s" ;
                MeanDopplerVelocity:resolution = 0.001f ;
                MeanDopplerVelocity:missing_value = -9999.f ;
        float SignalToNoiseRatio(time, heights) ;
                SignalToNoiseRatio:long_name = "Signal to Noise Ratio" ;
                SignalToNoiseRatio:units = "dB" ;
                SignalToNoiseRatio:resolution = 0.001f ;
                SignalToNoiseRatio:missing_value = -9999.f ;
        float Power(time, heights) ;
                Power:long_name = "Power (uncalibrated)" ;
                Power:units = "dB" ;
                Power:resolution = 0.001f ;
                Power:missing_value = -9999.f ;
        float SpectralWidth(time, heights) ;
                SpectralWidth:long_name = "Spectral Width" ;
                SpectralWidth:units = "m/s" ;
                SpectralWidth:resolution = 0.001f ;
                SpectralWidth:missing_value = -9999.f ;
        float NoiseLevel(time, heights) ;
                NoiseLevel:long_name = "Mean Noise Level" ;
                NoiseLevel:units = "dB" ;
                NoiseLevel:resolution = 0.001f ;
                NoiseLevel:missing_value = -99999.f ;
        float Reflectivity(time, heights) ;
                Reflectivity:long_name = "Reflectivity" ;
                Reflectivity:units = "dBZ" ;
                Reflectivity:resolution = 0.001f ;
                Reflectivity:missing_value = -9999.f ;
                Reflectivity:comment1 = "Nearfield correction has been applied" ;
                Reflectivity:comment2 = "Reference: Sekelsky, S. (2002), JAOT, v19, no. 4" ;
        float RangeCorrectedPower(time, heights) ;
                RangeCorrectedPower:long_name = "Range Corrected Calibrated Power" ;
                RangeCorrectedPower:units = "dBm" ;
                RangeCorrectedPower:resolution = 0.001f ;
                RangeCorrectedPower:missing_value = -9999.f ;
        float CircularDepolarizationRatio(time, heights) ;
                CircularDepolarizationRatio:long_name = "Circular Depolarization Ratio" ;
                CircularDepolarizationRatio:units = "dB" ;
                CircularDepolarizationRatio:resolution = 0.001f ;
                CircularDepolarizationRatio:missing_value = -9999.f ;

 
decode_coords=False did not work for this case
Then we would need a procedure that looks for variables that have the same name as a dimension in a netCDF file. Then it would suggest a change to the user and save a new file for xarray compatibility.


Bobby


________________________________
From: Adam Theisen <notifications@github.com>
Sent: Wednesday, May 22, 2019 9:45:31 AM
To: ANL-DIGR/ACT
Cc: Jackson, Robert; Comment
Subject: Re: [ANL-DIGR/ACT] Xarray not able to read old MMCR data (#67)


netcdf sgpmmcrmomC1.b1.20041107.000014 {
dimensions:
time = UNLIMITED ; // (44959 currently)
mode = 10 ;
namelength = 64 ;
heights = 167 ;
hourly = 24 ;
variables:
int base_time ;
base_time:string = "7-Nov-2004,0:00:14 GMT" ;
base_time:long_name = "Base time in Epoch" ;
base_time:units = "seconds since 1970-1-1 0:00:00 0:00" ;
double time_offset(time) ;
time_offset:long_name = "Time offset from base_time" ;
time_offset:units = "seconds since 2004-11-07 00:00:14 0:00" ;
double time(time) ;
time:long_name = "Time offset from midnight" ;
time:units = "seconds since 2004-11-07 00:00:00 0:00" ;

    float heights(mode, heights) ;
            heights:long_name = "Range Heights (center of radar sample volume)" ;
            heights:units = "m MSL" ;
            heights:missing_value = -9999.f ;
    int DataQualityStatus(time) ;
            DataQualityStatus:long_name = "Data Quality Status" ;
            DataQualityStatus:units = "code" ;
            DataQualityStatus:missing_value = -9999 ;
    float MeanDopplerVelocity(time, heights) ;
            MeanDopplerVelocity:long_name = "Mean Doppler Velocity" ;
            MeanDopplerVelocity:units = "m/s" ;
            MeanDopplerVelocity:resolution = 0.001f ;
            MeanDopplerVelocity:missing_value = -9999.f ;
    float SignalToNoiseRatio(time, heights) ;
            SignalToNoiseRatio:long_name = "Signal to Noise Ratio" ;
            SignalToNoiseRatio:units = "dB" ;
            SignalToNoiseRatio:resolution = 0.001f ;
            SignalToNoiseRatio:missing_value = -9999.f ;
    float Power(time, heights) ;
            Power:long_name = "Power (uncalibrated)" ;
            Power:units = "dB" ;
            Power:resolution = 0.001f ;
            Power:missing_value = -9999.f ;
    float SpectralWidth(time, heights) ;
            SpectralWidth:long_name = "Spectral Width" ;
            SpectralWidth:units = "m/s" ;
            SpectralWidth:resolution = 0.001f ;
            SpectralWidth:missing_value = -9999.f ;
    float NoiseLevel(time, heights) ;
            NoiseLevel:long_name = "Mean Noise Level" ;
            NoiseLevel:units = "dB" ;
            NoiseLevel:resolution = 0.001f ;
            NoiseLevel:missing_value = -99999.f ;
    float Reflectivity(time, heights) ;
            Reflectivity:long_name = "Reflectivity" ;
            Reflectivity:units = "dBZ" ;
            Reflectivity:resolution = 0.001f ;
            Reflectivity:missing_value = -9999.f ;
            Reflectivity:comment1 = "Nearfield correction has been applied" ;
            Reflectivity:comment2 = "Reference: Sekelsky, S. (2002), JAOT, v19, no. 4" ;
    float RangeCorrectedPower(time, heights) ;
            RangeCorrectedPower:long_name = "Range Corrected Calibrated Power" ;
            RangeCorrectedPower:units = "dBm" ;
            RangeCorrectedPower:resolution = 0.001f ;
            RangeCorrectedPower:missing_value = -9999.f ;
    float CircularDepolarizationRatio(time, heights) ;
            CircularDepolarizationRatio:long_name = "Circular Depolarization Ratio" ;
            CircularDepolarizationRatio:units = "dB" ;
            CircularDepolarizationRatio:resolution = 0.001f ;
            CircularDepolarizationRatio:missing_value = -9999.f ;


—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<https://github.com/ANL-DIGR/ACT/issues/67?email_source=notifications&email_token=AFIQA5FQYYMDPA7MAQTAHYDPWVMAXA5CNFSM4HOU6TA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV7JHIA#issuecomment-494834592>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFIQA5GMZQJFE2RADGDTQZDPWVMAXANCNFSM4HOU6TAQ>.

Oh crud.. it puts all the modes in one file!

 

Bobby is right.. we effectively need a mode splitter

 

 

From: Bobby Jackson <notifications@github.com>
Reply-To: ANL-DIGR/ACT <reply@reply.github.com>
Date: Wednesday, May 22, 2019 at 9:51 AM
To: ANL-DIGR/ACT <ACT@noreply.github.com>
Cc: Scott <scollis.acrf@gmail.com>, Comment <comment@noreply.github.com>
Subject: Re: [ANL-DIGR/ACT] Xarray not able to read old MMCR data (#67)

 

Then we would need a procedure that looks for variables that have the same name as a dimension in a netCDF file. Then it would suggest a change to the user and save a new file for xarray compatibility.


Bobby


________________________________
From: Adam Theisen <notifications@github.com>
Sent: Wednesday, May 22, 2019 9:45:31 AM
To: ANL-DIGR/ACT
Cc: Jackson, Robert; Comment
Subject: Re: [ANL-DIGR/ACT] Xarray not able to read old MMCR data (#67)


netcdf sgpmmcrmomC1.b1.20041107.000014 {
dimensions:
time = UNLIMITED ; // (44959 currently)
mode = 10 ;
namelength = 64 ;
heights = 167 ;
hourly = 24 ;
variables:
int base_time ;
base_time:string = "7-Nov-2004,0:00:14 GMT" ;
base_time:long_name = "Base time in Epoch" ;
base_time:units = "seconds since 1970-1-1 0:00:00 0:00" ;
double time_offset(time) ;
time_offset:long_name = "Time offset from base_time" ;
time_offset:units = "seconds since 2004-11-07 00:00:14 0:00" ;
double time(time) ;
time:long_name = "Time offset from midnight" ;
time:units = "seconds since 2004-11-07 00:00:00 0:00" ;

float heights(mode, heights) ;
heights:long_name = "Range Heights (center of radar sample volume)" ;
heights:units = "m MSL" ;
heights:missing_value = -9999.f ;
int DataQualityStatus(time) ;
DataQualityStatus:long_name = "Data Quality Status" ;
DataQualityStatus:units = "code" ;
DataQualityStatus:missing_value = -9999 ;
float MeanDopplerVelocity(time, heights) ;
MeanDopplerVelocity:long_name = "Mean Doppler Velocity" ;
MeanDopplerVelocity:units = "m/s" ;
MeanDopplerVelocity:resolution = 0.001f ;
MeanDopplerVelocity:missing_value = -9999.f ;
float SignalToNoiseRatio(time, heights) ;
SignalToNoiseRatio:long_name = "Signal to Noise Ratio" ;
SignalToNoiseRatio:units = "dB" ;
SignalToNoiseRatio:resolution = 0.001f ;
SignalToNoiseRatio:missing_value = -9999.f ;
float Power(time, heights) ;
Power:long_name = "Power (uncalibrated)" ;
Power:units = "dB" ;
Power:resolution = 0.001f ;
Power:missing_value = -9999.f ;
float SpectralWidth(time, heights) ;
SpectralWidth:long_name = "Spectral Width" ;
SpectralWidth:units = "m/s" ;
SpectralWidth:resolution = 0.001f ;
SpectralWidth:missing_value = -9999.f ;
float NoiseLevel(time, heights) ;
NoiseLevel:long_name = "Mean Noise Level" ;
NoiseLevel:units = "dB" ;
NoiseLevel:resolution = 0.001f ;
NoiseLevel:missing_value = -99999.f ;
float Reflectivity(time, heights) ;
Reflectivity:long_name = "Reflectivity" ;
Reflectivity:units = "dBZ" ;
Reflectivity:resolution = 0.001f ;
Reflectivity:missing_value = -9999.f ;
Reflectivity:comment1 = "Nearfield correction has been applied" ;
Reflectivity:comment2 = "Reference: Sekelsky, S. (2002), JAOT, v19, no. 4" ;
float RangeCorrectedPower(time, heights) ;
RangeCorrectedPower:long_name = "Range Corrected Calibrated Power" ;
RangeCorrectedPower:units = "dBm" ;
RangeCorrectedPower:resolution = 0.001f ;
RangeCorrectedPower:missing_value = -9999.f ;
float CircularDepolarizationRatio(time, heights) ;
CircularDepolarizationRatio:long_name = "Circular Depolarization Ratio" ;
CircularDepolarizationRatio:units = "dB" ;
CircularDepolarizationRatio:resolution = 0.001f ;
CircularDepolarizationRatio:missing_value = -9999.f ;


—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<https://github.com/ANL-DIGR/ACT/issues/67?email_source=notifications&email_token=AFIQA5FQYYMDPA7MAQTAHYDPWVMAXA5CNFSM4HOU6TA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV7JHIA#issuecomment-494834592>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFIQA5GMZQJFE2RADGDTQZDPWVMAXANCNFSM4HOU6TAQ>.


—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.


Yep, this was the old fashion way of handling modes.  Interleaved in the data.  Mode varied with time and you had to pull out the modes.  Fun times
I think I have a method down here that uses the NetCDF4 library.  The only question then is if we want to expand the function to breakout the modes.

Resolving patch:
diff --git a/act/io/__init__.py b/act/io/__init__.py
--- a/act/io/__init__.py
+++ b/act/io/__init__.py
@@ -14,6 +14,7 @@
             'check_arm_standards',
             'create_obj_from_arm_dod',
             'read_netcdf',
+            'read_mmcr',
         ],
         'csvfiles': ['read_csv'],
         'icartt': ['read_icartt'],
diff --git a/act/io/armfiles.py b/act/io/armfiles.py
--- a/act/io/armfiles.py
+++ b/act/io/armfiles.py
@@ -76,7 +76,7 @@ def read_netcdf(
 
     Returns
     -------
-    act_obj : Object (or None)
+    obj : Object (or None)
         ACT dataset (or None if no data file(s) found).
 
     Examples
@@ -298,7 +298,7 @@ def keep_variables_to_drop_variables(
 
     Returns
     -------
-    act_obj : list of str
+    obj : list of str
         Variable names to exclude from returned Dataset by using drop_variables keyword
         when calling Xarray.open_dataset().
 
@@ -736,3 +736,78 @@ def write_netcdf(
                 pass
 
         write_obj.to_netcdf(encoding=encoding, **kwargs)
+
+def read_mmcr(filenames):
+    """
+
+    Reads in ARM MMCR files and splits up the variables into specific
+    mode variables based on what's in the files.  MMCR files have the modes
+    interleaved and are not readable using xarray so some modifications are
+    needed ahead of time.
+
+    Parameters
+    ----------
+    filenames : str, pathlib.PosixPath or list of str
+        Name of file(s) to read.
+
+    Returns
+    -------
+    obj : Object (or None)
+        ACT dataset (or None if no data file(s) found).
+
+    """
+
+    # Sort the files to make sure they concatenate right
+    filenames.sort()
+
+    # Run through each file and read it in using netCDF4, then
+    # read it in with xarray
+    objs = []
+    for f in filenames:
+        nc = Dataset(f, "a")
+        # Change heights name to range to read appropriately to xarray
+        if 'heights' in nc.dimensions:
+            nc.renameDimension('heights', 'range')
+        if nc is not None:
+            obj = xr.open_dataset(xr.backends.NetCDF4DataStore(nc))
+            objs.append(obj)
+    # Concatenate objects together
+    if len(objs) > 1:
+        obj = xr.concat(objs, dim='time')
+    else:
+        obj = objs
+
+    # Get mdoes and ranges with time/height modes
+    modes = obj['mode'].values
+    mode_vars = []
+    for v in obj:
+        if 'range' in obj[v].dims and 'time' in obj[v].dims and len(obj[v].dims) == 2:
+            mode_vars.append(v)
+
+    # For each mode, run extract data variables if available
+    # saves as individual variables in the file.
+    for m in modes:
+        mode_desc = obj['ModeDescription'].values[0, m]
+        if np.isnan(obj['heights'].values[0, m, :]).all():
+            continue
+        mode_desc = str(mode_desc).split('_')[-1][0:-1]
+        mode_desc = str(mode_desc).split('\'')[0]
+        idx = np.where(obj['ModeNum'].values == m)[0]
+        range_data = obj['heights'].values[0, m, :]
+        idy = np.where(~np.isnan(range_data))[0]
+        for v in mode_vars:
+            new_var_name = v + '_' + mode_desc
+            time_name = 'time_' + mode_desc
+            range_name = 'range_' + mode_desc
+            data = obj[v].values[idx, :]
+            data = data[:, idy]
+            attrs = obj[v].attrs
+            da = xr.DataArray(
+                data=data,
+                coords={time_name: obj['time'].values[idx], range_name: range_data[idy]},
+                dims=[time_name, range_name],
+                attrs=attrs
+            )
+            obj[new_var_name] = da
+
+    return obj
diff --git a/act/plotting/timeseriesdisplay.py b/act/plotting/timeseriesdisplay.py
--- a/act/plotting/timeseriesdisplay.py
+++ b/act/plotting/timeseriesdisplay.py
@@ -296,6 +296,10 @@ def plot(
         Makes a timeseries plot. If subplots have not been added yet, an axis
         will be created assuming that there is only going to be one plot.
 
+        If plotting a high data volume 2D dataset, it may take some time to plot.
+        In order to speed up your plot creation, please resample your data to a
+        lower resolution dataset.
+
         Parameters
         ----------
         field : str


Test patch:
diff --git a/act/tests/test_io.py b/act/tests/test_io.py
--- a/act/tests/test_io.py
+++ b/act/tests/test_io.py
@@ -1,10 +1,9 @@
 import glob
 import tempfile
-from pathlib import Path
-
+import os
 import numpy as np
 import pytest
-
+from pathlib import Path
 import act
 import act.tests.sample_files as sample_files
 from act.io import read_gml, read_psl_wind_profiler_temperature, icartt
@@ -591,3 +590,28 @@ def test_read_icartt():
     assert 'Revision' in result.attrs
     np.testing.assert_almost_equal(
         result['static_pressure'].mean(), 708.75, decimal=2)
+
+
+def test_read_mmcr():
+    # Place your username and token here
+    username = os.getenv('ARM_USERNAME')
+    token = os.getenv('ARM_PASSWORD')
+    if not os.path.isdir(os.getcwd() + '/data/'):
+        os.makedirs(os.getcwd() + '/data/')
+    outdir = os.getcwd() + '/data/'
+    if username is not None and token is not None:
+        if len(username) == 0 and len(token) == 0:
+            return
+        datastream = 'sgpmmcrmomC1.b1'
+        startdate = '2009-01-01'
+        enddate = '2009-01-03'
+        results = act.discovery.get_armfiles.download_data(
+            username, token, datastream, startdate, enddate, output=outdir
+        )
+        obj = act.io.armfiles.read_mmcr(results)
+        assert 'MeanDopplerVelocity_PR' in obj
+        assert 'SpectralWidth_BL' in obj
+        np.testing.assert_almost_equal(
+            obj['Reflectivity_GE'].mean(), -33.96, decimal=2)
+        np.testing.assert_almost_equal(
+            obj['MeanDopplerVelocity_Receiver1'].max(), 10.14, decimal=2)