Repository: ARM-DOE/ACT

Issue:
Handling Incorrect ARM DQRs when applied with ACT function
We have implemented a function to query the ARM Data Quality Report database to return time periods when data is flagged. There are three levels of flagging within the DQRs, with two of them able to replace the variable values with NaN. ARM has a lot of DQRs where the list of variables is not accurate, and is over listing the affected variables.

For example a DQR may indicate the instrument is malfunctioning and select all variables in the netCDF file. While the data is incorrect the location, time, and QC variables are correct. Currently we loop over variables and apply the DQR to variables listed in the DQR. Some variables listed are scalars which is causing an exception that we catch and handle. Handling the exception has a print statement indicating the scalar variable could not be indexed.

We should discuss a better way to handle these variables as the print statement may confuse users when there is nothing wrong with the scalar variables. A couple of options include:

1) Correctly handling the replacement of data values with NaN for the scalar variables
2) Setting a keyword to enact the print statement, but set to False as default
3) Skip over these most likely incorrectly listed variables.

An example output for sgpaosccn2colaE13.b1 on 20170903 using D170905.3
```
Skipping 'eta_lookup_table' DQR application because of IndexError
Skipping 'base_time' DQR application because of IndexError
Skipping 'alt' DQR application because of IndexError
Skipping 'droplet_size_bounds' DQR application because of IndexError
Skipping 'lon' DQR application because of IndexError
Skipping 'lat' DQR application because of IndexError
```


Discussion:

Resolving patch:
diff --git a/act/qc/arm.py b/act/qc/arm.py
--- a/act/qc/arm.py
+++ b/act/qc/arm.py
@@ -20,6 +20,7 @@ def add_dqr_to_qc(
     normalize_assessment=True,
     cleanup_qc=True,
     dqr_link=False,
+    skip_location_vars=False,
 ):
     """
     Function to query the ARM DQR web service for reports and
@@ -63,6 +64,9 @@ def add_dqr_to_qc(
         if the Dataset has already been cleaned up, no need to run.
     dqr_link : boolean
         Prints out a link for each DQR to read the full DQR.  Defaults to False
+    skip_location_vars : boolean
+        Does not apply DQRs to location variables.  This can be useful in the event
+        the submitter has erroneously selected all variables.
 
     Returns
     -------
@@ -104,7 +108,11 @@ def add_dqr_to_qc(
         variable = [variable]
 
     # Loop through each variable and call web service for that variable
+    loc_vars = ['lat', 'lon', 'alt', 'latitude', 'longitude', 'altitude']
     for var_name in variable:
+        if skip_location_vars:
+            if var_name in loc_vars:
+                continue
         # Create URL
         url = 'http://www.archive.arm.gov/dqrws/ARMDQR?datastream='
         url += datastream
@@ -146,9 +154,13 @@ def add_dqr_to_qc(
             starttime = np.datetime64(dt.datetime.utcfromtimestamp(int(line[1])))
             endtime = np.datetime64(dt.datetime.utcfromtimestamp(int(line[2])))
             ind = np.where((time >= starttime) & (time <= endtime))
+
             if ind[0].size == 0:
                 continue
 
+            if 'time' not in ds[var_name].dims:
+                ind = np.where((ds[var_name].values == ds[var_name].values) | (np.isnan(ds[var_name].values)))
+
             if dqr_no in dqr_results.keys():
                 dqr_results[dqr_no]['index'] = np.append(dqr_results[dqr_no]['index'], ind)
             else:


Test patch: