internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: RedHatInsights/insights-core

Issue:
Need to add the Yum updates datasource to the documentation
The [PR](https://github.com/RedHatInsights/insights-core/pull/2993/files#diff-22151ef794ba196097984a47bf24b6759c261de6dc062ac541da099084e5c50a) adding this datasource did not add the datasource to the documentation [here](https://github.com/RedHatInsights/insights-core/blob/master/docs/custom_datasources_index.rst).


Discussion:

Resolving patch:
diff --git a/insights/specs/datasources/yum_updates.py b/insights/specs/datasources/yum_updates.py
--- a/insights/specs/datasources/yum_updates.py
+++ b/insights/specs/datasources/yum_updates.py
@@ -70,7 +70,7 @@ def installed_packages(self):
     def updates(self, pkg):
         nevra = pkg.nevra
         updates_list = []
-        for upg in self.updict[pkg.na]:
+        for upg in self.updict.get(pkg.na, []):
             if upg.verGT(pkg):
                 updates_list.append(upg)
         return nevra, updates_list
@@ -100,6 +100,32 @@ def yum_updates(_broker):
     This datasource provides a list of available updates on the system.
     It uses the yum python library installed locally, and collects list of
     available package updates, along with advisory info where applicable.
+
+    Sample data returned::
+
+        {
+          "releasever": "8",
+          "basearch": "x86_64",
+          "update_list": {
+            "NetworkManager-1:1.22.8-4.el8.x86_64": {
+              "available_updates": [
+                {
+                  "package": "NetworkManager-1:1.22.8-5.el8_2.x86_64",
+                  "repository": "rhel-8-for-x86_64-baseos-rpms",
+                  "basearch": "x86_64",
+                  "releasever": "8",
+                  "erratum": "RHSA-2020:3011"
+                }
+              ]
+            }
+          },
+          "metadata_time": "2021-01-01T09:39:45Z"
+        }
+
+    Returns:
+        list: List of available updates
+    Raises:
+        SkipComponent: Raised on systems different than RHEL 7
     """
 
     if not _broker.get(IsRhel7):


Test patch:
Repository: RedHatInsights/insights-core

Issue:
Update pydoc for Messages (/var/log/messages)
Correct the examples: https://github.com/RedHatInsights/insights-core/blob/master/insights/parsers/messages.py#L32


Discussion:

Resolving patch:
diff --git a/insights/parsers/messages.py b/insights/parsers/messages.py
--- a/insights/parsers/messages.py
+++ b/insights/parsers/messages.py
@@ -30,17 +30,14 @@ class Messages(Syslog):
         This will also work around December/January crossovers.
 
     Examples:
-        >>> Messages.filters.append('wrapper')
         >>> Messages.token_scan('daemon_start', 'Wrapper Started as Daemon')
-        >>> msgs = shared[Messages]
+        >>> type(msgs)
+        <class 'insights.parsers.messages.Messages'>
         >>> len(msgs.lines)
+        9
         >>> wrapper_msgs = msgs.get('wrapper') # Can only rely on lines filtered being present
-        >>> wrapper_msgs[0]
-        {'timestamp': 'May 18 15:13:36', 'hostname': 'lxc-rhel68-sat56',
-         'procname': wrapper[11375]', 'message': '--> Wrapper Started as Daemon',
-         'raw_message': 'May 18 15:13:36 lxc-rhel68-sat56 wrapper[11375]: --> Wrapper Started as Daemon'
-        }
-        >>> msgs.daemon_start # Token set if matching lines present in logs
+        >>> result = {'raw_message': 'May 18 15:13:36 lxc-rhel68-sat56 wrapper[11375]: --> Wrapper Started as Daemon', 'message': '--> Wrapper Started as Daemon', 'timestamp': 'May 18 15:13:36', 'hostname': 'lxc-rhel68-sat56', 'procname': 'wrapper[11375]'}
+        >>> wrapper_msgs[0] == result
         True
     """
     pass


Test patch:
diff --git a/insights/parsers/tests/test_messages.py b/insights/parsers/tests/test_messages.py
--- a/insights/parsers/tests/test_messages.py
+++ b/insights/parsers/tests/test_messages.py
@@ -1,5 +1,7 @@
+import doctest
+
 from insights import add_filter
-from insights.parsers.messages import Messages
+from insights.parsers import messages
 from insights.specs import Specs
 from insights.tests import context_wrap
 
@@ -25,8 +27,17 @@
 ])
 
 
+def test_doc_examples():
+    env = {
+        'msgs': messages.Messages(context_wrap(MSGINFO)),
+        'Messages': messages.Messages
+    }
+    failed, total = doctest.testmod(messages, globs=env)
+    assert failed == 0
+
+
 def test_messages():
-    msg_info = Messages(context_wrap(MSGINFO))
+    msg_info = messages.Messages(context_wrap(MSGINFO))
     bona_list = msg_info.get('(root) LIST (root)')
     assert 2 == len(bona_list)
     assert bona_list[0].get('timestamp') == "Apr 22 10:37:32"
Repository: RedHatInsights/insights-core

Issue:
Insights core skips rules if test is in the package name
Review the fix made in [Core PR 3209](https://github.com/RedHatInsights/insights-core/pull/3209) to exclude insights core tests. The fix now causes rules to be skipped that contain 'test' anywhere in the package name. The check should only look for 'test' in the beginning of the file name so that only actual 'test' components will be eliminated.


Discussion:

Resolving patch:
diff --git a/insights/core/dr.py b/insights/core/dr.py
--- a/insights/core/dr.py
+++ b/insights/core/dr.py
@@ -393,7 +393,7 @@ def _import(path, continue_on_error):
             raise
 
 
-def _load_components(path, include=".*", exclude="test", continue_on_error=True):
+def _load_components(path, include=".*", exclude="insights\\..+\\.tests", continue_on_error=True):
     do_include = re.compile(include).search if include else lambda x: True
     do_exclude = re.compile(exclude).search if exclude else lambda x: False
 


Test patch:
Repository: RedHatInsights/insights-core

Issue:
Exception occurs in ConfigCombiner when the main conf doesn't exist.
This was reproduced by installing nginx then moving /etc/nginx/nginx.conf to /etc/nginx/nginx.conf.bak, but it could occur with any combiner using ConfigCombiner and the main_file argument doesn't exist in confs.

Traceback (most recent call last):
  File "/var/lib/insights/last_stable.egg/insights/core/dr.py", line 967, in run
    result = DELEGATES[component].process(broker)
  File "/var/lib/insights/last_stable.egg/insights/core/dr.py", line 686, in process
    return self.invoke(broker)
  File "/var/lib/insights/last_stable.egg/insights/core/plugins.py", line 64, in invoke
    return super(PluginType, self).invoke(broker)
  File "/var/lib/insights/last_stable.egg/insights/core/dr.py", line 666, in invoke
    return self.component(*args)
  File "/var/lib/insights/last_stable.egg/insights/combiners/nginx_conf.py", line 73, in __init__
    super(NginxConfTree, self).__init__(confs, "nginx.conf", eq("include"))
  File "/var/lib/insights/last_stable.egg/insights/core/__init__.py", line 374, in __init__
    server_root = self.conf_path
  File "/var/lib/insights/last_stable.egg/insights/combiners/nginx_conf.py", line 77, in conf_path
    return os.path.dirname(self.main.file_path)
AttributeError: 'NoneType' object has no attribute 'file_path'


Discussion:

Resolving patch:
diff --git a/insights/core/__init__.py b/insights/core/__init__.py
--- a/insights/core/__init__.py
+++ b/insights/core/__init__.py
@@ -129,12 +129,6 @@ def default_parser_deserializer(_type, data):
     return obj
 
 
-def find_main(confs, name):
-    for c in confs:
-        if c.file_name == name:
-            return c
-
-
 def flatten(docs, pred):
     """
     Replace include nodes with their config trees.  Allows the same files to be
@@ -370,7 +364,7 @@ class ConfigCombiner(ConfigComponent):
     """
     def __init__(self, confs, main_file, include_finder):
         self.confs = confs
-        self.main = find_main(confs, main_file)
+        self.main = self.find_main(main_file)
         server_root = self.conf_path
 
         # Set the children of all include directives to the contents of the
@@ -391,6 +385,13 @@ def find_matches(self, confs, pattern):
         results = [c for c in confs if fnmatch(c.file_path, pattern)]
         return sorted(results, key=operator.attrgetter("file_name"))
 
+    def find_main(self, name):
+        for c in self.confs:
+            if c.file_name == name:
+                return c
+
+        raise SkipException("The main conf {main_conf} doesn't exist.".format(main_conf=name))
+
 
 class LegacyItemAccess(object):
     """


Test patch:
Repository: RedHatInsights/insights-core

Issue:
Pinned jinja2 version is no longer supported and conflicts with the latest MarkupSafe
**Current behavior**

insights-core has pinned the jinja2 version to <=2.11.3 in #3057. This version is [no longer supported](https://github.com/pallets/jinja/issues/1585#issuecomment-1043892628).

**Expected behavior**

insights-core can be used with a maintained jinja2 version in applications that depend both on insights-core and jinja2.

**Impact**

jinja2-2.11.3 fails to import with the latest version of MarkupSafe, 2.1.0:

```
...
.tox/unit-py38/lib/python3.8/site-packages/jinja2/filters.py:13: in <module>
    from markupsafe import soft_unicode
E   ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/builds/ccx/ccx-rules-ocp/.tox/unit-py38/lib64/python3.8/site-packages/markupsafe/__init__.py
```

All projects that depend on insights-core have to pin unsupported versions of MarkupSafe now. I would argue that it is important to keep up-to-date for a package that is used for secure handling of user input.


Discussion:
Btw. I am unable to figure why the jinja2 version has been pinned.

With Python 3, all tests pass even with both
* jinja2-3.0.0 and markupsafe-2.1.0
* jinja2-3.0.3 and markupsafe-2.1.0

With Python 2, pip pulls in jinja2-2.11.3 and markupsafe-1.1.1 even if the versions are not pinned in setup.py. All tests pass, kind of.

#3057 doesn't explain what the error was...

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -32,7 +32,8 @@
     'cachecontrol[filecache]',
     'defusedxml',
     'lockfile',
-    'jinja2<=2.11.3',
+    'jinja2<=2.11.3; python_version <= "2.7"',
+    'jinja2; python_version > "2.7"',
     'pyyaml>=3.10,<=3.13; python_version < "2.7"',
     'pyyaml; python_version >= "2.7"',
 ])
@@ -66,7 +67,6 @@ def maybe_require(pkg):
     'sphinx_rtd_theme',
     'ipython',
     'colorama',
-    'jinja2<=2.11.3',
     'Pygments',
     'jedi<0.18.0',    # Open issue with jedi 0.18.0 and iPython <= 7.19
                       # https://github.com/davidhalter/jedi/issues/1714


Test patch:
Repository: RedHatInsights/insights-core

Issue:
When invoking collect cli, args are being parsed by client and failing
When running `insights collect` or `python insights/collect.py` the malware app spec is invoking client configuration and args passed to collect are being parsed by the client.  This causes an exit directly out of collection due to the argparse error.  Args should not be parsed beyond collect.py when collect is invoked from the command line.

```
$ python insights/collect.py -o /tmp -v
...
INFO:insights.core.dr:Trying insights.specs.datasources.malware_detection.MalwareDetectionSpecs.malware_detection_app
usage: collect.py [-h] [--ansible-host ANSIBLE_HOST] [--checkin]
                  [--collector APP] [--compliance] [--conf CONF]
                  [--disable-schedule] [--display-name DISPLAY_NAME]
                  [--enable-schedule] [--group GROUP] [--keep-archive]
                  [--list-specs] [--logging-file LOGGING_FILE] [--net-debug]
                  [--no-upload] [--offline] [--output-dir OUTPUT_DIR]
                  [--output-file OUTPUT_FILE] [--quiet] [--register]
                  [--force-reregister] [--retry RETRIES] [--show-results]
                  [--silent] [--status] [--support] [--test-connection]
                  [--unregister] [--validate] [--verbose] [--version]
                  [--payload PAYLOAD] [--content-type CONTENT_TYPE]
                  [--diagnosis [DIAGNOSIS]]
collect.py: error: ambiguous option: -o could match -od, -of
```


Discussion:

Resolving patch:
diff --git a/insights/collect.py b/insights/collect.py
--- a/insights/collect.py
+++ b/insights/collect.py
@@ -11,6 +11,7 @@
 import argparse
 import logging
 import os
+import sys
 import tempfile
 import yaml
 
@@ -416,6 +417,12 @@ def collect(manifest=default_manifest, tmp_path=None, compress=False, rm_conf=No
 
 
 def main():
+    # Remove command line args so that they are not parsed by any called modules
+    # The main fxn is only invoked as a cli, if calling from another cli then
+    # use the collect function instead
+    collect_args = [arg for arg in sys.argv[1:]] if len(sys.argv) > 1 else []
+    sys.argv = [sys.argv[0], ] if sys.argv else sys.argv
+
     p = argparse.ArgumentParser()
     p.add_argument("-m", "--manifest", help="Manifest yaml.")
     p.add_argument("-o", "--out_path", help="Path to write output data.")
@@ -423,7 +430,7 @@ def main():
     p.add_argument("-v", "--verbose", help="Verbose output.", action="store_true")
     p.add_argument("-d", "--debug", help="Debug output.", action="store_true")
     p.add_argument("-c", "--compress", help="Compress", action="store_true")
-    args = p.parse_args()
+    args = p.parse_args(args=collect_args)
 
     level = logging.WARNING
     if args.verbose:


Test patch:
Repository: RedHatInsights/insights-core

Issue:
Use absolute path to timeout command
When using `timeout` to ensure commands executed for collection don't run too long, make sure we use the absolute path to the executable.


Discussion:
The [current code](https://github.com/RedHatInsights/insights-core/blob/master/insights/util/subproc.py#L80) uses the `which` utility function to get the full path to the `timeout` command.

Resolving patch:
diff --git a/insights/client/insights_spec.py b/insights/client/insights_spec.py
--- a/insights/client/insights_spec.py
+++ b/insights/client/insights_spec.py
@@ -8,7 +8,7 @@
 import sys
 from subprocess import Popen, PIPE, STDOUT
 from tempfile import NamedTemporaryFile
-from insights.util import mangle
+from insights.util import mangle, which
 
 from .constants import InsightsConstants as constants
 from .utilities import determine_hostname
@@ -61,13 +61,15 @@ def get_output(self):
         else:
             signal = 'KILL'
 
-        timeout_command = 'timeout -s %s %s %s' % (
-            signal, self.config.cmd_timeout, self.command)
-
         # ensure consistent locale for collected command output
         cmd_env = {'LC_ALL': 'C',
                    'PATH': '/sbin:/bin:/usr/sbin:/usr/bin',
                    'PYTHONPATH': os.getenv('PYTHONPATH')}
+
+        timeout = which('timeout', env=cmd_env)
+        timeout_command = '%s -s %s %s %s' % (
+            timeout, signal, self.config.cmd_timeout, self.command)
+
         args = shlex.split(timeout_command)
 
         # never execute this stuff


Test patch:
Repository: googleapis/google-api-python-client

Issue:
HttpMock response content should be bytes but isn't (in Python 3)
Consider the following test code, to mock a 400 Bad Request error (as returned if the API key is invalid):

``` python
http = HttpMock('bad_request.json', {'status': '400'})
response = some_api_call(http=http)
```

The expected behaviour here would be raising an `HttpError`, however on Python 3 this results in a mysterious `TypeError: HTTP content should be bytes` bubbling up from the `HttpError` constructor.

My investigations led me to discover that:
- HTTP response content should always be bytes (confirmed [in a related pull request](https://github.com/google/google-api-python-client/pull/81#r27543602))
- `HttpMock` reads response content from files in text mode, so it gets a string rather than bytes on Python 3
- `HttpError` contains the only `isinstance(..., bytes)` check in the entire library, so testing with non-error status codes yields working tests
- 100% of test cases using `HttpMock` use status=200

As far as I can tell, this is an extremely simple fix: `HttpMock` should read the file in `rb` mode. (Simple) pull request with test and fix to follow.



Discussion:

Resolving patch:
diff --git a/googleapiclient/http.py b/googleapiclient/http.py
--- a/googleapiclient/http.py
+++ b/googleapiclient/http.py
@@ -1463,7 +1463,7 @@ def __init__(self, filename=None, headers=None):
     if headers is None:
       headers = {'status': '200'}
     if filename:
-      f = open(filename, 'r')
+      f = open(filename, 'rb')
       self.data = f.read()
       f.close()
     else:


Test patch:
diff --git a/tests/data/bad_request.json b/tests/data/bad_request.json
new file mode 100644
--- /dev/null
+++ b/tests/data/bad_request.json
@@ -0,0 +1,13 @@
+{
+ "error": {
+  "errors": [
+   {
+    "domain": "usageLimits",
+    "reason": "keyInvalid",
+    "message": "Bad Request"
+   }
+  ],
+  "code": 400,
+  "message": "Bad Request"
+ }
+}
diff --git a/tests/test_http.py b/tests/test_http.py
--- a/tests/test_http.py
+++ b/tests/test_http.py
@@ -1010,6 +1010,17 @@ def test_default_response_headers(self):
     resp, content = http.request("http://example.com")
     self.assertEqual(resp.status, 200)
 
+  def test_error_response(self):
+    http = HttpMock(datafile('bad_request.json'), {'status': '400'})
+    model = JsonModel()
+    request = HttpRequest(
+        http,
+        model.response,
+        'https://www.googleapis.com/someapi/v1/collection/?foo=bar',
+        method='GET',
+        headers={})
+    self.assertRaises(HttpError, request.execute)
+
 
 if __name__ == '__main__':
   logging.getLogger().setLevel(logging.ERROR)
Repository: googleapis/google-api-python-client

Issue:
Error importing discovery from the client API
Hello, I'm getting a name exception when running a simple script. I've tried running the script in two ways:

```
from apiclient import discovery
import json

from pprint import pprint
from httplib2 import Http

from oauth2client.service_account import ServiceAccountCredentials

scopes = ['https://www.googleapis.com/auth/fusiontables']

credentials = ServiceAccountCredentials.from_json_keyfile_name(
    'service-account.json', scopes)

ft = discovery.build('fusiontables', 'v2', credentials=credentials)
pprint(ft)
```

Which gives:

> Traceback (most recent call last):
>   File "apiTest.py", line 1, in <module>
>     from apiclient import discovery
>   File "build/bdist.linux-x86_64/egg/apiclient/**init**.py", line 18, in <module>
>   File "build/bdist.linux-x86_64/egg/googleapiclient/channel.py", line 64, in <module>
> ImportError: cannot import name util

And:

```
from apiclient.discovery import build
import json

from pprint import pprint
from httplib2 import Http

from oauth2client.service_account import ServiceAccountCredentials

scopes = ['https://www.googleapis.com/auth/fusiontables']

credentials = ServiceAccountCredentials.from_json_keyfile_name(
    'service-account.json', scopes)

ft = discovery.build('fusiontables', 'v2', credentials=credentials)
pprint(ft)
#print response
```

Which yields the same exception:

> python apiTest.py 
> Traceback (most recent call last):
>   File "apiTest.py", line 1, in <module>
>     from apiclient.discovery import build
>   File "build/bdist.linux-x86_64/egg/apiclient/**init**.py", line 18, in <module>
>   File "build/bdist.linux-x86_64/egg/googleapiclient/channel.py", line 64, in <module>
> ImportError: cannot import name util

I'm not sure if I'm missing a prereq or what, but I'm not able to find a way around this.



Discussion:
What version are you on? This seems like the oauth2client compatibility issue that was fixed in 1.5.2.

I'm on 1.5.2.

In fact, I just cloned in today.

What version of oauth2client are you using?

oh no, I just realized I missed one: https://github.com/google/google-api-python-client/blob/master/googleapiclient/channel.py#L64

That line just needs to be removed. I'll try to get a PR and new release in this week.

Thanks so much!


Resolving patch:
diff --git a/googleapiclient/channel.py b/googleapiclient/channel.py
--- a/googleapiclient/channel.py
+++ b/googleapiclient/channel.py
@@ -61,7 +61,6 @@
 import uuid
 
 from googleapiclient import errors
-from oauth2client import util
 import six
 
 # Oauth2client < 3 has the positional helper in 'util', >= 3 has it


Test patch:
Repository: googleapis/google-api-python-client

Issue:
Switch from "uritemplate" to "uritemplate.py"
There are at least 2 URI template packages on the PyPI: [uritemplate](https://pypi.python.org/pypi/uritemplate/0.6) and [uritemplate.py](https://pypi.python.org/pypi/uritemplate.py/0.3.0). Both of these packages use `uritemplate` as the package name to be imported, which causes a conflict when both are installed (see https://github.com/sigmavirus24/uritemplate/issues/14).

I propose that this project depend on  `uritemplate.py` because:
- `uritemplate.py` is API-compatible with `uritemplate`, but the reverse is not true. I have confirmed that this library works with uritemplate.py.
- Other projects depend on `uritemplate.py`. For example, we are currently working on a project that depends on both `github3.py` (GitHub API client) and `google-api-python-client`. Installing both results in an immediate `ImportError` due to the `uritemplate` conflict.

This  is a simple, low-risk change that would aid compatibility with other projects.

Thoughts?



Discussion:
+1     We are having the same issue of incompatibility between the 'github3.py' and 'google-api-python-client' libraries.  

Switching this project over to using `uritemplate.py` would make life much easier.

Even though the technical aspect of the change sounds very simple I'd like to know more before welcoming a pull request welcoming the change. This smells like a political problem, like an unclean fork. What's being done upstream between uritemplate and uritemplate.py to resolve the problem?

Any update on this issue? Discogs is trying to use this client for the YouTube v3 API but we are running into this same error.

This hasn't gone away. As to the upstream, it seems that uritemplate.py claims API compatibility with  uritemplate which hasn't had an update since Aug 27, 2013. While it may be a political issue the effect is very real, running an ansible playbook that needs to talk to google cloud and github at the same time is... challenging 

For folks subscribed to this issue but not otherwise watching the project: #146 just went by and I didn't realize during its review that this issue was opened on the matter. Has this issue budged at all in the last few months? I see recent commits over at https://github.com/uri-templates/uritemplate-py/commits/master including one by @graingert.

For everyone affected by this issue: has anyone ever bothered to contact the maintainers of both libraries and simply ask what's up? It's been nearly three years since either of them cut a release; I wonder if they can be merged and the whole problem solved properly.


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -66,7 +66,7 @@ def _DetectBadness():
     'httplib2>=0.8,<1',
     'oauth2client>=1.5.0,<4.0.0',
     'six>=1.6.1,<2',
-    'uritemplate>=0.6,<1',
+    'uritemplate>=3.0.0,<4',
 ]
 
 if sys.version_info < (2, 7):


Test patch: