internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ansible/ansible-lint

Issue:
missing include causes traceback
Not existing includes cause a traceback instead of a warning:

```

---
  pre_tasks:
    - { include: 'doesnotexist.yml'}
```

traceback

```
Traceback (most recent call last):
  File "/usr/local/bin/ansible-lint", line 110, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/bin/ansible-lint", line 97, in main
    matches = runner.run()
  File "/usr/local/lib/python2.7/dist-packages/ansiblelint/__init__.py", line 180, in run
    skip_list=set(self.skip_list)))
  File "/usr/local/lib/python2.7/dist-packages/ansiblelint/__init__.py", line 106, in run
    with open(playbookfile['path'], 'Ur') as f:
IOError: [Errno 2] No such file or directory: u'doesnotexist.yml'
```

I'm using test-kitchen and the file does exist in the vm, but not on the host, fixed it with a softlink to self, but I think it should not traceback but report a error.



Discussion:

Resolving patch:
diff --git a/lib/ansiblelint/__init__.py b/lib/ansiblelint/__init__.py
--- a/lib/ansiblelint/__init__.py
+++ b/lib/ansiblelint/__init__.py
@@ -18,8 +18,10 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 from collections import defaultdict
 import os
+import sys
 
 import ansiblelint.utils
 
@@ -112,8 +114,16 @@ def extend(self, more):
     def run(self, playbookfile, tags=set(), skip_list=set()):
         text = ""
         matches = list()
-        with open(playbookfile['path'], 'Ur') as f:
-            text = f.read()
+
+        try:
+            with open(playbookfile['path'], 'Ur') as f:
+                text = f.read()
+        except IOError, e:
+            print("WARNING: Couldn't open %s - %s" %
+                  (playbookfile['path'], e.strerror),
+                  file=sys.stderr)
+            return matches
+
         for rule in self.rules:
             if not tags or not set(rule.tags).isdisjoint(tags):
                 rule_definition = set(rule.tags)


Test patch:
diff --git a/test/block.yml b/test/block.yml
--- a/test/block.yml
+++ b/test/block.yml
@@ -1,6 +1,9 @@
 ---
 - hosts: all
 
+  pre_tasks:
+  - { include: 'doesnotexist.yml' }
+
   tasks:
   - block:
       - debug: msg='i execute normally'
Repository: ansible/ansible-lint

Issue:
Wrong format specification
In one of may playbooks i had an error, a task including a playbook file, which should have triggered a RuntimeException from ansible-lint, but the code had a parenthesis in the wrong place which throws a TypeError instead. Below is a patch for it.

``` diff
diff --git a/lib/ansiblelint/utils.py b/lib/ansiblelint/utils.py
index 258d0b6..e0b5cd3 100644
--- a/lib/ansiblelint/utils.py
+++ b/lib/ansiblelint/utils.py
@@ -277,7 +277,7 @@ def normalize_task(task):

                     else:
                         # Should not get here!
-                        raise RuntimeError("Was not expecting value %s of type %s for key %s\nTask: %s" % (str(v), type(v), k), str(task))  # noqa
+                        raise RuntimeError("Was not expecting value %s of type %s for key %s\nTask: %s" % (str(v), type(v), k, str(task)))  # noqa
             v['module_arguments'] = v.get('module_arguments', list())
             result['action'] = v
     return result
```



Discussion:
Hi, thanks for that. I can definitely get that fixed. However, the comment says it shouldn't get there, so there's clearly another bug.

Are you able to include a playbook that triggers this error?

I forgot. Sorry.

``` yaml
# main.yml
- hosts: all
  tasks:
    - include: test.yml

# test.yml
- hosts: all
  tasks:
    - debug: msg="Hello World!"
```

Thanks, I'll take a look!


Resolving patch:
diff --git a/lib/ansiblelint/utils.py b/lib/ansiblelint/utils.py
--- a/lib/ansiblelint/utils.py
+++ b/lib/ansiblelint/utils.py
@@ -276,8 +276,12 @@ def normalize_task(task):
                         continue
 
                     else:
-                        # Should not get here!
-                        raise RuntimeError("Was not expecting value %s of type %s for key %s\nTask: %s" % (str(v), type(v), k), str(task))  # noqa
+                        # Tasks that include playbooks (rather than task files)
+                        # can get here
+                        # https://github.com/willthames/ansible-lint/issues/138
+                        raise RuntimeError("Was not expecting value %s of type %s for key %s\nTask: %s. "
+                                           "Check the syntax of your playbook using ansible-playbook "
+                                           "--syntax-check" % (str(v), type(v), k, str(task)))
             v['module_arguments'] = v.get('module_arguments', list())
             result['action'] = v
     return result


Test patch:
Repository: ansible/ansible-lint

Issue:
ansible-lint ignores -x parameters except the last one
# Issue Type
- Bug report

# Ansible and Ansible Lint details

ansible-2.3.2.0 & ansible-lint-3.4.15, installed via pip

# Desired Behaviour

`-x foo,bar -x baz` should result in all of `foo`, `bar` and `baz` being ignored.

# Actual Behaviour (Bug report only)

`-x foo,bar -x baz` results in only `baz` being ignored.

See [this build](https://travis-ci.org/hashbang/admin-tools/builds/272718491#L493) vs. [this one](https://travis-ci.org/hashbang/admin-tools/builds/272720279#L493)


Discussion:
ping

Resolving patch:
diff --git a/lib/ansiblelint/__init__.py b/lib/ansiblelint/__init__.py
--- a/lib/ansiblelint/__init__.py
+++ b/lib/ansiblelint/__init__.py
@@ -117,7 +117,7 @@ def __len__(self):
     def extend(self, more):
         self.rules.extend(more)
 
-    def run(self, playbookfile, tags=set(), skip_list=set()):
+    def run(self, playbookfile, tags=set(), skip_list=frozenset()):
         text = ""
         matches = list()
 
@@ -240,7 +240,7 @@ def run(self):
             if self.verbosity > 0:
                 print("Examining %s of type %s" % (file['path'], file['type']))
             matches.extend(self.rules.run(file, tags=set(self.tags),
-                           skip_list=set(self.skip_list)))
+                           skip_list=self.skip_list))
         # update list of checked files
         self.checked_files.update([x['path'] for x in files])
 
diff --git a/lib/ansiblelint/__main__.py b/lib/ansiblelint/__main__.py
--- a/lib/ansiblelint/__main__.py
+++ b/lib/ansiblelint/__main__.py
@@ -87,7 +87,7 @@ def main():
     parser.add_option('-v', dest='verbosity', action='count',
                       help="Increase verbosity level",
                       default=0)
-    parser.add_option('-x', dest='skip_list', default=[],
+    parser.add_option('-x', dest='skip_list', default=[], action='append',
                       help="only check rules whose id/tags do not " +
                       "match these values")
     parser.add_option('--nocolor', dest='colored',
@@ -160,8 +160,11 @@ def main():
 
     if isinstance(options.tags, six.string_types):
         options.tags = options.tags.split(',')
-    if isinstance(options.skip_list, six.string_types):
-        options.skip_list = options.skip_list.split(',')
+
+    skip = set()
+    for s in options.skip_list:
+         skip.add(*s.split(','))
+    options.skip_list = frozenset(skip)
 
     playbooks = set(args)
     matches = list()


Test patch:
Repository: ARM-DOE/ACT

Issue:
Add Google Analytics ID
Add a Google Analytics ID to the `conf.py` file used by sphinx. For those interested in having access the analytics, you will need to send over your gmail address

Fixes #396 


Discussion:

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -126,7 +126,9 @@
 # further.  For a list of options available for each theme, see the
 # documentation.
 #
-# html_theme_options = {}
+html_theme_options = {
+    'google_analytics_id': 'UA-179020619-3',
+}
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,


Test patch:
Repository: ARM-DOE/ACT

Issue:
PSL Wind Profile Reader Hardcoded to CTD site
Line 38 in io.noaapsl.read_psl_wind_profiler is hardcoded to the CTD site, we should automatically read the site code in and parse off of that or update to pass in a site name.


Discussion:
@zssherman would you want to take a stab at this one?
@AdamTheisen Yeah I can take a look!

Resolving patch:
diff --git a/act/io/noaapsl.py b/act/io/noaapsl.py
--- a/act/io/noaapsl.py
+++ b/act/io/noaapsl.py
@@ -34,8 +34,11 @@ def read_psl_wind_profiler(filename, transpose=True):
     # read file with pandas for preparation.
     df = pd.read_csv(filename, header=None)
 
+    # The first entry should be the station identifier (ex. CTD)
+    potential_site = df[0][0]
+
     # Get location of where each table begins
-    index_list = df[0] == ' CTD'
+    index_list = df[0] == potential_site
     idx = np.where(index_list)
 
     # Get header of each column of data.


Test patch:
Repository: ARM-DOE/ACT

Issue:
ARM Data API Returns Text when File not Available 
I tried to download gucceilpblhtM1.a0 but it's not available through the webservice due to it being an a0-level file.  Instead the API returned text that was included in the file.  We should put in a check to ensure that these cases are caught and files are not produced.

```
This data file is not available on /data/archive.  To download this file, please an order via Data Discovery. https://adc.arm.gov/discovery
```


Discussion:
@AdamTheisen So files produced, so you want this instead to catch the warning, warn the user and just return from the function without writing anything?
@zssherman yes, if the data are not available, we wouldn't want to write the files and would instead return the message as an error.  Right now, it just gives the false sense that it's downloading files
Gotcha sounds good
@AdamTheisen So for the datetime i chose for that dataset it returned 2 netcdf files. Are these the files you mentioned? I did not get that message though or is the message in these files?
Nvm yep found it!

Resolving patch:
diff --git a/act/discovery/get_armfiles.py b/act/discovery/get_armfiles.py
--- a/act/discovery/get_armfiles.py
+++ b/act/discovery/get_armfiles.py
@@ -4,6 +4,7 @@
 """
 
 import argparse
+import contextlib
 import json
 import os
 import sys
@@ -143,17 +144,28 @@ def download_data(username, token, datastream, startdate, enddate, time=None, ou
             if time is not None:
                 if time not in fname:
                     continue
-            print(f'[DOWNLOADING] {fname}')
             # construct link to web service saveData function
             save_data_url = (
                 'https://adc.arm.gov/armlive/livedata/' + 'saveData?user={0}&file={1}'
             ).format(':'.join([username, token]), fname)
             output_file = os.path.join(output_dir, fname)
+
+            # if datastream is not in data archive, return error
+            with contextlib.closing(urlopen(save_data_url).read().decode().lower()) as check:
+                if 'this data file is not available' in check:
+                    raise OSError(
+                        "The datastream '" + datastream
+                        + "' is not available on /data/archive. To download "
+                        + "this file, place an order via Data Discovery. "
+                        + "https://adc.arm.gov/discovery")
+
             # make directory if it doesn't exist
             if not os.path.isdir(output_dir):
                 os.makedirs(output_dir)
+
             # create file and write bytes to file
             with open(output_file, 'wb') as open_bytes_file:
+                print(f'[DOWNLOADING] {fname}')
                 open_bytes_file.write(urlopen(save_data_url).read())
             file_names.append(output_file)
     else:


Test patch:
Repository: ARM-DOE/ACT

Issue:
Accessor not available in dataset
I fetched the latest updates after the lazy_loading PR and ran pytest and am seeing a lot of errors with accessors not loading.  Clean, QCFilter, and QCTests are no longer available in the datasets for some reason.

FAILED test_io.py::test_io_mfdataset - AttributeError: 'Dataset' object has no attribute 'clean'
FAILED test_io.py::test_io_write - AttributeError: 'Dataset' object has no attribute 'clean'
FAILED test_io.py::test_clean_cf_qc - AttributeError: 'Dataset' object has no attribute 'clean'


Discussion:
@AdamTheisen can you provide a code snippet to reproduce the error
@mgrover1 see below.  I went to the ACT/act/__init__.py file and added in `from .qc import clean, QCFilter, QCTests` and that fixed the issues I was seeing but not sure that's the right way.

```
import act
sonde_ds = act.io.armfiles.read_netcdf([act.tests.EXAMPLE_MET1])
sonde_ds.clean.cleanup()
```

The other option is to just run pytest test_io.py in the test area

Resolving patch:
diff --git a/act/__init__.py b/act/__init__.py
--- a/act/__init__.py
+++ b/act/__init__.py
@@ -5,11 +5,13 @@
 """
 
 import lazy_loader as lazy
+
 # No more pandas warnings
 from pandas.plotting import register_matplotlib_converters
 
 from . import tests
 from ._version import get_versions
+from .qc import QCFilter, QCTests, clean
 
 register_matplotlib_converters()
 
diff --git a/act/qc/__init__.py b/act/qc/__init__.py
--- a/act/qc/__init__.py
+++ b/act/qc/__init__.py
@@ -8,6 +8,8 @@
 
 # We need to import clean first to register the accessor
 from .clean import *
+from .qcfilter import QCFilter
+from .qctests import QCTests
 
 __getattr__, __dir__, __all__ = lazy.attach(
     __name__,


Test patch:
Repository: ARM-DOE/ACT

Issue:
ICARTT formatting error when using cleanup
When I read in an ICARTT file and then try and run obj.clean.cleanup() to get it to CF standards, it throws the following error.  We should look at making the object compliant with what's needed for this cleanup.

```
Traceback (most recent call last):
  File "/Users/atheisen/opt/anaconda3/lib/python3.8/site-packages/xarray/core/dataset.py", line 1395, in _construct_dataarray
    variable = self._variables[name]
KeyError: 'flag'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "plot_icartt.py", line 22, in <module>
    ds.clean.cleanup()
  File "/Users/atheisen/Code/ACT/act/qc/clean.py", line 151, in cleanup
    self._obj.clean.link_variables()
  File "/Users/atheisen/Code/ACT/act/qc/clean.py", line 583, in link_variables
    self._obj[variable].attrs['ancillary_variables'] = copy.copy(ancillary_variables)
  File "/Users/atheisen/opt/anaconda3/lib/python3.8/site-packages/xarray/core/dataset.py", line 1499, in __getitem__
    return self._construct_dataarray(key)
  File "/Users/atheisen/opt/anaconda3/lib/python3.8/site-packages/xarray/core/dataset.py", line 1397, in _construct_dataarray
    _, name, variable = _get_virtual_variable(
  File "/Users/atheisen/opt/anaconda3/lib/python3.8/site-packages/xarray/core/dataset.py", line 170, in _get_virtual_variable
    ref_var = variables[ref_name]
KeyError: 'flag'
```


Discussion:
Can you provide an example dataset?
@kenkehoe  I was just using the sample_file in the test area (EXAMPLE_AAF_ICARTT) but now I just tried this and it's saying that the object does not have a an attribute "clean" so I think something got a little off with recent updates here.

Resolving patch:
diff --git a/act/io/icartt.py b/act/io/icartt.py
--- a/act/io/icartt.py
+++ b/act/io/icartt.py
@@ -103,9 +103,13 @@ def read_icartt(filename, format=_format,
         # Note time is the only independent variable within ICARTT
         # Short name for time must be "Start_UTC" for ICARTT files.
         if key != 'Start_UTC':
+            if key == 'qc_flag':
+                key2 = 'quality_flag'
+            else:
+                key2 = key
             da = xr.DataArray(ict.data[key],
                               coords=dict(time=ict.times),
-                              name=key, dims=['time'])
+                              name=key2, dims=['time'])
             # Assume if Uncertainity does not match the number of variables,
             # values were not set within the file. Needs to be string!
             if len(uncertainty) != len(ict.variables):
@@ -133,7 +137,7 @@ def read_icartt(filename, format=_format,
             da.attrs['ULOD_Flag'] = ulod_flag
             da.attrs['LLOD_Flag'] = llod_flag
             # Append to ds container
-            ds_container.append(da.to_dataset(name=key))
+            ds_container.append(da.to_dataset(name=key2))
             # up the counter
             counter += 1
 


Test patch:
Repository: ARM-DOE/ACT

Issue:
Additional plotting examples
Some additional needs were brought up with the ACT plotting examples.  It would be useful to provide

- Examples for working with axes returned from plotting code and modifying them after plot generation
- Examples for using the integrated xarray plotting for quick plots


Discussion:

Resolving patch:
diff --git a/examples/plot_examples.py b/examples/plot_examples.py
new file mode 100644
--- /dev/null
+++ b/examples/plot_examples.py
@@ -0,0 +1,41 @@
+"""
+Xarray Plotting Examples
+------------------------
+
+This is an example of how to use some different aspects
+of ACT's plotting tools as well as Xarray's tools.
+
+"""
+
+import matplotlib.pyplot as plt
+import xarray as xr
+import act
+
+
+# Set up plot space ahead of time
+fig, ax = plt.subplots(3, figsize=(10, 7))
+
+# Plotting up high-temporal resolution 2D data can be very slow at times.
+# In order to increase the speed, the data can be resampled to a courser
+# resolution prior to plotting.  Using Xarray's resample and selecting
+# the nearest neighbor will greatly increase the speed.
+obj = act.io.armfiles.read_netcdf(act.tests.EXAMPLE_CEIL1)
+obj = obj.resample(time='1min').nearest()
+
+# These data can be plotted up using the existing xarray functionality
+# which is quick and easy
+obj['backscatter'].plot(x='time', ax=ax[0])
+
+# or using ACT
+display = act.plotting.TimeSeriesDisplay(obj)
+display.assign_to_figure_axis(fig, ax[1])
+display.plot('backscatter')
+
+# When using ACT, the axis object can also be manipulated using normal
+# matplotlib calls for more personalized customizations
+display = act.plotting.TimeSeriesDisplay(obj)
+display.assign_to_figure_axis(fig, ax[2])
+display.plot('backscatter')
+display.axes[-1].set_ylim([0, 1500])
+
+plt.show()


Test patch:
Repository: ARM-DOE/ACT

Issue:
Valid Min in act.utils.decode_present_weather
* ACT version: 1.3.4
* Python version:3.9.15
* Operating System: Windows 10

### Description

I am receiving an error for "del data.attrs['valid_min'] when reading in data trying to decode present weather data. 

### What I Did

```
act.utils.decode_present_weather(ds_fd70,'present_wx2',precip_codes)
decode_present_weather
    del data.attrs['valid_min']

KeyError: 'valid_min'

When outputting time I have:
ds_fd70.time

array(['2023-03-09T16:45:00.000000000', '2023-03-09T17:00:00.000000000',
       '2023-03-09T17:15:00.000000000', '2023-03-09T17:30:00.000000000',
       '2023-03-09T17:45:00.000000000', '2023-03-09T18:00:00.000000000',
       '2023-03-09T18:15:00.000000000', '2023-03-09T18:30:00.000000000',
       '2023-03-09T18:45:00.000000000', '2023-03-09T19:00:00.000000000',
       '2023-03-09T19:15:00.000000000', '2023-03-09T19:30:00.000000000',
```



Discussion:

Resolving patch:
diff --git a/act/utils/inst_utils.py b/act/utils/inst_utils.py
--- a/act/utils/inst_utils.py
+++ b/act/utils/inst_utils.py
@@ -134,9 +134,14 @@ def decode_present_weather(ds, variable=None, decoded_name=None):
 
     # Massage the data array to set back in the dataset
     data.values = wx_type
-    data.attrs['long_name'] = data.attrs['long_name'] + ' Decoded'
-    del data.attrs['valid_min']
-    del data.attrs['valid_max']
+    if 'long_name' in data.attrs:
+        data.attrs['long_name'] = data.attrs['long_name'] + ' Decoded'
+    else:
+        data.attrs['long_name'] = 'Decoded present weather values'
+    if 'valid_min' in data.attrs:
+        del data.attrs['valid_min']
+    if 'valid_max' in data.attrs:
+        del data.attrs['valid_max']
 
     ds[decoded_name] = data
 


Test patch:
Repository: ARM-DOE/ACT

Issue:
Example Incorrect
### Description

This example says it's for plotting sounding data but the example uses MET data.  We should update this to be what it was originally intended.
https://arm-doe.github.io/ACT/source/auto_examples/plotting/plot_sonde.html




Discussion:

Resolving patch:
diff --git a/examples/plotting/plot_sonde.py b/examples/plotting/plot_sonde.py
--- a/examples/plotting/plot_sonde.py
+++ b/examples/plotting/plot_sonde.py
@@ -12,20 +12,14 @@
 
 import act
 
-files = act.tests.sample_files.EXAMPLE_MET_WILDCARD
-met_ds = act.io.armfiles.read_netcdf(files)
-print(met_ds)
-met_temp = met_ds.temp_mean
-met_rh = met_ds.rh_mean
-met_lcl = (20.0 + met_temp / 5.0) * (100.0 - met_rh) / 1000.0
-met_ds['met_lcl'] = met_lcl * 1000.0
-met_ds['met_lcl'].attrs['units'] = 'm'
-met_ds['met_lcl'].attrs['long_name'] = 'LCL Calculated from SGP MET E13'
+files = act.tests.sample_files.EXAMPLE_SONDE1
+sonde_ds = act.io.armfiles.read_netcdf(files)
+print(sonde_ds)
 
 # Plot data
-display = act.plotting.TimeSeriesDisplay(met_ds)
+display = act.plotting.TimeSeriesDisplay(sonde_ds)
 display.add_subplots((3,), figsize=(15, 10))
-display.plot('wspd_vec_mean', subplot_index=(0,))
-display.plot('temp_mean', subplot_index=(1,))
-display.plot('rh_mean', subplot_index=(2,))
+display.plot('wspd', subplot_index=(0,))
+display.plot('tdry', subplot_index=(1,))
+display.plot('rh', subplot_index=(2,))
 plt.show()


Test patch:
diff --git a/act/tests/test_retrievals.py b/act/tests/test_retrievals.py
--- a/act/tests/test_retrievals.py
+++ b/act/tests/test_retrievals.py
@@ -252,7 +252,7 @@ def test_sp2_waveform_stats():
     my_binary = act.qc.get_waveform_statistics(my_sp2b, my_ini, parallel=False)
     assert my_binary.PkHt_ch1.max() == 62669.4
     np.testing.assert_almost_equal(np.nanmax(my_binary.PkHt_ch0.values), 98708.92915295, decimal=1)
-    np.testing.assert_almost_equal(np.nanmax(my_binary.PkHt_ch4.values), 54734.05714286, decimal=1)
+    np.testing.assert_almost_equal(np.nanmax(my_binary.PkHt_ch4.values), 65088.39598033, decimal=1)
 
 
 @pytest.mark.skipif(not PYSP2_AVAILABLE, reason="PySP2 is not installed.")