internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: biolab/orange3-text

Issue:
Preprocess text widget should remove stop-words
<!--
This is an issue template. Please fill in the relevant details in the
sections below.
-->

##### Text version
<!-- From menu _Options→Add-ons→Orange3-Text_ or code `orangecontrib.text.version.full_version` -->
0.7.3dev

##### Orange version
<!-- From menu _Help→About→Version_ or code `Orange.version.full_version` -->
3.24.0dev

##### Expected behavior

- Preprocess text widget should remove stop-words from the text for the Slovene language.
- Stopwords languages should be ordered in alphabetical order.

##### Actual behavior

It does not happen when stopwords from NLTK are used - the problem is spaces beside the stopwords.

##### Steps to reproduce the behavior

Read some data with Import documents and connect with Preprocess Text widget. Observe the output of Preprocess Text widget.

##### Additional info (worksheets, data, screenshots, ...)





Discussion:

Resolving patch:
diff --git a/orangecontrib/text/preprocess/filter.py b/orangecontrib/text/preprocess/filter.py
--- a/orangecontrib/text/preprocess/filter.py
+++ b/orangecontrib/text/preprocess/filter.py
@@ -78,7 +78,7 @@ def supported_languages():
         except LookupError:  # when no NLTK data is available
             pass
 
-        return [file.capitalize() for file in stopwords_listdir]
+        return sorted(file.capitalize() for file in stopwords_listdir)
 
     @wait_nltk_data
     def __init__(self, language='English', word_list=None):
@@ -96,7 +96,8 @@ def language(self, value):
         if not self._language:
             self.stopwords = []
         else:
-            self.stopwords = set(stopwords.words(self.language.lower()))
+            self.stopwords = set(
+                x.strip() for x in stopwords.words(self.language.lower()))
 
     def __str__(self):
         config = ''


Test patch:
diff --git a/orangecontrib/text/tests/test_preprocess.py b/orangecontrib/text/tests/test_preprocess.py
--- a/orangecontrib/text/tests/test_preprocess.py
+++ b/orangecontrib/text/tests/test_preprocess.py
@@ -276,10 +276,24 @@ def check(self, token):
         self.assertEqual(df([['a', '1']]), [['a']])
 
     def test_stopwords(self):
-        filter = preprocess.StopwordsFilter('english')
+        f = preprocess.StopwordsFilter('english')
 
-        self.assertFalse(filter.check('a'))
-        self.assertTrue(filter.check('filter'))
+        self.assertFalse(f.check('a'))
+        self.assertTrue(f.check('filter'))
+
+        self.assertListEqual(
+            ["snake", "house"],
+            f(["a", "snake", "is", "in", "a", "house"]))
+
+    def test_stopwords_slovene(self):
+        f = preprocess.StopwordsFilter('slovene')
+
+        self.assertFalse(f.check('in'))
+        self.assertTrue(f.check('abeceda'))
+
+        self.assertListEqual(
+            ["kača", "hiši"],
+            f(["kača", "je", "v", "hiši", "in"]))
 
     def test_lexicon(self):
         filter = preprocess.LexiconFilter(['filter'])
Repository: biolab/orange3-text

Issue:
Word Cloud: annoying white line at the top
Word Cloud plot has two bugs.
1.) The visualization is not always immediately plotted and one needs to press 'Regenerate word cloud' for anything to be displayed.
2.) The annoying small line of mirrored text when the visualization is resized.


Discussion:
Haven't encountered 1. in a while. 2. is still an issue.
Can you add more descriptions, possibly with screenshots? In particular, which white line at the top?

I cannot replicate either bug. I cannot even resize the window; the program crashed with "Segmentation fault: 11" error.
Segementation fault is due to a bad version of PyQt. I recommend installing PyQt5 = 5.12.3.

Here is the screenshot:
<img width="659" alt="Screen Shot 2019-10-14 at 09 31 28" src="https://user-images.githubusercontent.com/12524972/66735088-0c331b80-ee6e-11e9-8898-d2d3f068dd91.png">

The issue is that two images get drawn at the same time, so when as WC is full (lots of words), you might even see a bit of the words on the top line.

Resolving patch:
diff --git a/orangecontrib/text/widgets/owwordcloud.py b/orangecontrib/text/widgets/owwordcloud.py
--- a/orangecontrib/text/widgets/owwordcloud.py
+++ b/orangecontrib/text/widgets/owwordcloud.py
@@ -88,7 +88,6 @@ def update_selection(self, words):
         self.mainArea.layout().addWidget(webview)
 
     def _create_layout(self):
-        self._new_webview()
         box = gui.widgetBox(self.controlArea, 'Info')
         self.topic_info = gui.label(box, self, '%(n_topic_words)d words in a topic')
         gui.label(box, self, '%(documents_info_str)s')


Test patch:
Repository: biolab/orange3-text

Issue:
Concordance: set Concordance as text feature + add meta data
Concordance widget has a Concordance output, which currently doesn't set Concordances as a text feature. This should not be so - set it as text. Also provide additional meta data to the output (whatever is in the input data set).


Discussion:

Resolving patch:
diff --git a/orangecontrib/text/widgets/owconcordance.py b/orangecontrib/text/widgets/owconcordance.py
--- a/orangecontrib/text/widgets/owconcordance.py
+++ b/orangecontrib/text/widgets/owconcordance.py
@@ -167,7 +167,7 @@ def get_data(self):
             data.append([" ".join(txt)])
             docs.append([self.corpus.titles[self.word_index[row][0]]])
         conc = np.array(np.hstack((data, docs)), dtype=object)
-        return Corpus(domain, metas=conc, text_features=[domain.metas[1]])
+        return Corpus(domain, metas=conc, text_features=[domain.metas[0]])
 
 
 class OWConcordance(OWWidget):


Test patch:
diff --git a/orangecontrib/text/widgets/tests/test_owconcordances.py b/orangecontrib/text/widgets/tests/test_owconcordances.py
--- a/orangecontrib/text/widgets/tests/test_owconcordances.py
+++ b/orangecontrib/text/widgets/tests/test_owconcordances.py
@@ -141,7 +141,10 @@ def test_concordance_output(self):
         model = ConcordanceModel()
         model.set_word("of")
         model.set_corpus(self.corpus)
-        self.assertEqual(len(model.get_data()), 7)
+        output = model.get_data()
+        self.assertEqual(len(output), 7)
+        self.assertEqual(len(output.text_features), 1)
+        self.assertTrue(output.text_features[0].name.startswith('Conc. of'))
 
 
 class TestConcordanceWidget(WidgetTest):
Repository: biolab/orange3-text

Issue:
Word Enrichment: sort by p-value
<!--
This is an issue template. Please fill in the relevant details in the
sections below.
-->

##### Text version
<!-- From menu _Options→Add-ons→Orange3-Text_ or code `orangecontrib.text.version.full_version` -->
0.8.0

##### Orange version
<!-- From menu _Help→About→Version_ or code `Orange.version.full_version` -->
3.26.0.dev

##### Expected behavior
Word Enrichment sorts by p-value by default.


##### Actual behavior
It sorts by words (alphabetically).





Discussion:
@ajdapretnar The widget is currently configured to sort by FDR in ascending order. I couldn't reproduce the case where it sorts alphabetically by default.
🤔 It probably should sort by p-values rather than FDR? The issue I have with FDR is that it is so often 1 for small data sets.
In that case it really should sort by p-value. 

Resolving patch:
diff --git a/orangecontrib/text/widgets/owwordenrichment.py b/orangecontrib/text/widgets/owwordenrichment.py
--- a/orangecontrib/text/widgets/owwordenrichment.py
+++ b/orangecontrib/text/widgets/owwordenrichment.py
@@ -115,7 +115,7 @@ def __init__(self):
         self.sig_words.setHeaderLabels(self.cols)
         self.sig_words.setSortingEnabled(True)
         self.sig_words.setSelectionMode(QTreeView.NoSelection)
-        self.sig_words.sortByColumn(2, 0)   # 0 is ascending order
+        self.sig_words.sortByColumn(1, 0)   # 0 is ascending order
         for i in range(len(self.cols)):
             self.sig_words.resizeColumnToContents(i)
         self.mainArea.layout().addWidget(self.sig_words)


Test patch:
Repository: sosreport/sos

Issue:
sosreport does not collect apt specific configuration on Debian/Ubuntu
Nothing from /etc/apt is collected on apt-based operating systems



Discussion:

Resolving patch:
diff --git a/sos/plugins/apt.py b/sos/plugins/apt.py
new file mode 100644
--- /dev/null
+++ b/sos/plugins/apt.py
@@ -0,0 +1,31 @@
+# Copyright (C) 2013 Louis Bouchard <louis.bouchard@ubuntu.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+
+from sos.plugins import Plugin, UbuntuPlugin, DebianPlugin
+
+class Apt(Plugin, DebianPlugin, UbuntuPlugin):
+    """ Apt Plugin
+    """
+
+    plugin_name = 'apt'
+
+    def setup(self):
+        self.add_copy_specs(["/etc/apt"])
+
+        self.add_cmd_output("apt-get check")
+        self.add_cmd_output("apt-config dump")
+        self.add_cmd_output("apt-cache stats")
+        self.add_cmd_output("apt-cache policy")


Test patch:
Repository: sosreport/sos

Issue:
neutron plugin does not have check_enabled
The neutron plugin is runs on non-OpenStack systems, because it has no check_enabled method like the other openstack plugins.

I'm happy to help (say the word), but this plugin only has the one commit and I'm not sure what it's status is.

Also, it gets the following exception because it is running (which it shouldn't) and it is assuming that network namespaces are enabled.

```
Traceback (most recent call last):
  File "/opt/sosreport/embedded/lib/python2.7/site-packages/sos/sosreport.py", line 940, in setup
    plug.setup()
  File "/opt/sosreport/embedded/lib/python2.7/site-packages/sos/plugins/neutron.py", line 170, in setup
    super(RedHatNeutron, self).setup()
  File "/opt/sosreport/embedded/lib/python2.7/site-packages/sos/plugins/neutron.py", line 49, in setup
    self.netns_dumps()
  File "/opt/sosreport/embedded/lib/python2.7/site-packages/sos/plugins/neutron.py", line 80, in netns_dumps
    prefix, netid = nsname.split('-', 1)
ValueError: need more than 1 value to unpack
```

```
# ip netns
Object "netns" is unknown, try "ip help".
# cat /etc/redhat-release 
CentOS release 6.3 (Final)
```



Discussion:
Hey Jeff,

This is the first iteration of the neutron plugin and if you have spare cycles to include a check_enabled that would be excellent. Otherwise, I can get to it sometime this week.

Thanks for the report!
Adam


Resolving patch:
diff --git a/sos/plugins/neutron.py b/sos/plugins/neutron.py
--- a/sos/plugins/neutron.py
+++ b/sos/plugins/neutron.py
@@ -139,6 +139,9 @@ class DebianNeutron(Neutron, DebianPlugin, UbuntuPlugin):
                             'python-%(comp)s',
                             'python-%(comp)sclient']
 
+    def check_enabled(self):
+        return self.is_installed("%s-common" % self.component_name)
+
     def setup(self):
         super(DebianNeutron, self).setup()
         self.packages = self.gen_pkg_tuple(self.package_list_template)
@@ -166,6 +169,9 @@ class RedHatNeutron(Neutron, RedHatPlugin):
                              'python-%(comp)s',
                              'python-%(comp)sclient']
 
+    def check_enabled(self):
+        return self.is_installed("openstack-%s" % self.component_name)
+
     def setup(self):
         super(RedHatNeutron, self).setup()
         self.packages = self.gen_pkg_tuple(self.package_list_template)


Test patch:
Repository: sosreport/sos

Issue:
[dbus] plugin
... we should have one!

I noticed a few d-busy things in the general plugin. There was also a proposal at one point for a user-level dbus plugin.



Discussion:

Resolving patch:
diff --git a/sos/plugins/dbus.py b/sos/plugins/dbus.py
new file mode 100644
--- /dev/null
+++ b/sos/plugins/dbus.py
@@ -0,0 +1,30 @@
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+
+from sos.plugins import Plugin, RedHatPlugin, DebianPlugin, UbuntuPlugin
+
+
+class Dbus(Plugin, RedHatPlugin, DebianPlugin, UbuntuPlugin):
+    """dbus information"""
+
+    plugin_name = "dbus"
+
+    def setup(self):
+        self.add_copy_specs([
+            "/etc/dbus-1",
+            "/var/lib/dbus/machine-id"
+        ])
+
+
+# vim: et ts=4 sw=4
diff --git a/sos/plugins/general.py b/sos/plugins/general.py
--- a/sos/plugins/general.py
+++ b/sos/plugins/general.py
@@ -27,7 +27,6 @@ def setup(self):
             "/proc/stat",
             "/var/log/pm/suspend.log",
             "/etc/hostid",
-            "/var/lib/dbus/machine-id",
             "/etc/localtime",
             "/etc/os-release"
         ])


Test patch:
Repository: sosreport/sos

Issue:
[block] Don't use parted human readable output - rhbz #1183770
Changed the parted command to return data in sectors units
instead of human readable form.

Signed-off-by: Shane Bradley sbradley@redhat.com



Discussion:

Resolving patch:
diff --git a/sos/plugins/block.py b/sos/plugins/block.py
--- a/sos/plugins/block.py
+++ b/sos/plugins/block.py
@@ -46,7 +46,7 @@ def setup(self):
                 disk_path = os.path.join('/dev/', disk)
                 self.add_cmd_output([
                     "udevadm info -ap /sys/block/%s" % (disk),
-                    "parted -s %s print" % (disk_path),
+                    "parted -s %s unit s print" % (disk_path),
                     "fdisk -l %s" % disk_path
                 ])
 


Test patch:
Repository: sosreport/sos

Issue:
Remove hashlib import from sos/utilities.py since 
With the removal of the hashing functions from sos/utilities.py there is no need to import hashlib.  The hashing was removed in the following commit: https://github.com/sosreport/sos/commit/6038fdf8617319a13b0b42f3283ec2066d54b283

$ gendiff sos/ .org
diff -up sos/sos/utilities.py.org sos/sos/utilities.py
--- sos/sos/utilities.py.org    2016-01-12 09:26:50.865294298 -0500
+++ sos/sos/utilities.py    2016-01-12 09:26:58.959233725 -0500
@@ -18,7 +18,6 @@ import os
 import re
 import inspect
 from subprocess import Popen, PIPE, STDOUT
-import hashlib
 import logging
 import fnmatch
 import errno



Discussion:

Resolving patch:
diff --git a/sos/utilities.py b/sos/utilities.py
--- a/sos/utilities.py
+++ b/sos/utilities.py
@@ -18,7 +18,6 @@
 import re
 import inspect
 from subprocess import Popen, PIPE, STDOUT
-import hashlib
 import logging
 import fnmatch
 import errno


Test patch:
Repository: sosreport/sos

Issue:
[rabbitmq] does not elide password in /etc/rabbitmq.config
Hi,
Sosreport as in sos-3.2-36.eo7ost.1.noarch includes in `/etc/rabbitmq/rabbitmq.conf` `default_user` and `default_pass` without being hidden.

Regards,
Pablo



Discussion:
I guess you comment `rabbitmq` plugin. Could you please show examples of `default_user` and `default_pass` lines that should be obfuscated?

Is it required to hide both user and password, or just password?

Ommited the real ones, but:

```
 cat etc/rabbitmq/rabbitmq.config
% This file managed by Puppet
% Template Path: rabbitmq/templates/rabbitmq.config
[
  {rabbit, [
    {cluster_partition_handling, pause_minority},
    {tcp_listen_options, [binary,{packet, raw},
                                {reuseaddr, true},
                                {backlog, 128},
                                {nodelay, true},
                                {exit_on_close, false},
                                {keepalive, true}]},
    {default_user, <<"USER">>},
    {default_pass, <<"PASSWORD">>}
  ]},
  {kernel, [
    {inet_dist_listen_max, 35672},
    {inet_dist_listen_min, 35672}
  ]}
].
% EOF
```

`USER` and `PASSWORD` contained customer-sensitive data

We don't normally eliminate user name values (since they are generally not considered sensitive and may be relevant for support purposes).

What's the rationale for eliding them for `rabbitmq`?

Well, at least the password should be (IMHO)


Resolving patch:
diff --git a/sos/plugins/rabbitmq.py b/sos/plugins/rabbitmq.py
--- a/sos/plugins/rabbitmq.py
+++ b/sos/plugins/rabbitmq.py
@@ -60,5 +60,8 @@ def setup(self):
             "/var/log/containers/rabbitmq/*"
         ], sizelimit=self.get_option('log_size'))
 
+    def postproc(self):
+        self.do_file_sub("/etc/rabbitmq/rabbitmq.conf",
+                         r"(\s*default_pass\s*,\s*)\S+", r"\1<<***>>},")
 
 # vim: set et ts=4 sw=4 :


Test patch:
Repository: sosreport/sos

Issue:
[pulp] no obfuscation of proxy_password in /etc/pulp/server/plugins.conf.d/*.json files
Reproducer: `/etc/pulp/server/plugins.conf.d/iso_importer.json` with:

        "proxy_password": 'my_SECRET_password',

Then `sos report` does not obfuscate this password.


Discussion:

Resolving patch:
diff --git a/sos/report/plugins/pulp.py b/sos/report/plugins/pulp.py
--- a/sos/report/plugins/pulp.py
+++ b/sos/report/plugins/pulp.py
@@ -160,8 +160,8 @@ def postproc(self):
         # Now handle JSON-formatted data in the same /etc/pulp directory
         # structure. We use a different substitution string here to preserve
         # the file's JSON syntax.
-        jreg = r"(\s*\".*(passw|cred|token|secret).*\"\s*:\s*\")(.*)(\")"
-        repl = r"\1********\4"
+        jreg = r"(\s*\".*(passw|cred|token|secret).*\"\s*:(\s))(.*)(\w+)"
+        repl = r"\1********"
         self.do_path_regex_sub("/etc/pulp(.*)(.json$)", jreg, repl)
 
         # obfuscate SECRET_KEY = .. and 'PASSWORD': .. in dynaconf list output


Test patch:
Repository: sosreport/sos

Issue:
Typo in help output for sos collect 
Hello, 
I just wanted to let you know that there is a typo in the following line obtained in the output for sos collect --help on RHEL 8:

--clean, --mask Obfuscate sensistive information

It should be sensitive instead of sensistive . 

I couldn't really find which file generates this specific output so I can't open a pull request for the moment. 

Cheers,
Jeremy 


Discussion:
Good eye! For reference, this is in https://github.com/sosreport/sos/blob/main/sos/collector/__init__.py#L419 (and seems a copy-paste has it in https://github.com/sosreport/sos/blob/main/sos/report/__init__.py#L345 as well) if you want to submit a fix.
I will try again to do a pull request tomorrow in the evening but this time I will do it the right way :) . 

Resolving patch:
diff --git a/sos/collector/__init__.py b/sos/collector/__init__.py
--- a/sos/collector/__init__.py
+++ b/sos/collector/__init__.py
@@ -416,7 +416,7 @@ def add_parser_options(cls, parser):
         cleaner_grp.add_argument('--clean', '--cleaner', '--mask',
                                  dest='clean',
                                  default=False, action='store_true',
-                                 help='Obfuscate sensistive information')
+                                 help='Obfuscate sensitive information')
         cleaner_grp.add_argument('--keep-binary-files', default=False,
                                  action='store_true', dest='keep_binary_files',
                                  help='Keep unprocessable binary files in the '
diff --git a/sos/report/__init__.py b/sos/report/__init__.py
--- a/sos/report/__init__.py
+++ b/sos/report/__init__.py
@@ -342,7 +342,7 @@ def add_parser_options(cls, parser):
         cleaner_grp.add_argument('--clean', '--cleaner', '--mask',
                                  dest='clean',
                                  default=False, action='store_true',
-                                 help='Obfuscate sensistive information')
+                                 help='Obfuscate sensitive information')
         cleaner_grp.add_argument('--domains', dest='domains', default=[],
                                  action='extend',
                                  help='Additional domain names to obfuscate')


Test patch:
Repository: sosreport/sos

Issue:
URL for commercial support seems wrong
Just noticed that the vendor URL for commercial support is not working. Looks like the leading `www.` should be removed:

https://github.com/sosreport/sos/blob/31b991ef2b51856264c911018eddea8dd55b5718/sos/policies/distros/redhat.py#L41

```
% curl https://www.access.redhat.com/
curl: (6) Could not resolve host: www.access.redhat.com

```


Discussion:
Good point.

Would you like to open a PR for it or shall I do so? (whatever works for me)
I'll open the PR, I wasn't sure if I was missing something :sweat_smile: 

Resolving patch:
diff --git a/sos/policies/distros/redhat.py b/sos/policies/distros/redhat.py
--- a/sos/policies/distros/redhat.py
+++ b/sos/policies/distros/redhat.py
@@ -38,7 +38,7 @@ class RedHatPolicy(LinuxPolicy):
     vendor = "Red Hat"
     vendor_urls = [
         ('Distribution Website', 'https://www.redhat.com/'),
-        ('Commercial Support', 'https://www.access.redhat.com/')
+        ('Commercial Support', 'https://access.redhat.com/')
     ]
     _tmp_dir = "/var/tmp"
     _in_container = False


Test patch:
Repository: sosreport/sos

Issue:
Some MAAS config files missing from collection
Currently we're only collecting `/var/lib/maas/dhcp`, meaning that we're missing other key config files that would help with troubleshooting MAAS issues, e.g.,  `/var/lib/maas/http`. I'd suggest to add the below paths to be collected:

* /var/lib/maas/http/*
* /var/lib/maas/*.conf


Discussion:
Thank you for reporting this issue @al3jandrosg . Do you want to send a PR, or should I?
Do any of these new files to be collected have any password or secrets that must be obfuscated?
Hi @jcastill, thanks for the quick response! I'll try to submit a PR later today, but feel free to do it yourself if you like or I don't manage to do that by the end of the day. I'll check if any of the files need obfuscation and will take that into account accordingly.
Ack @al3jandrosg . Let me know if you have any questions about the process. 

Resolving patch:
diff --git a/sos/report/plugins/maas.py b/sos/report/plugins/maas.py
--- a/sos/report/plugins/maas.py
+++ b/sos/report/plugins/maas.py
@@ -87,6 +87,9 @@ def setup(self):
                 "/etc/squid-deb-proxy",
                 "/etc/maas",
                 "/var/lib/maas/dhcp*",
+                "/var/lib/maas/http/*.conf",
+                "/var/lib/maas/*.conf",
+                "/var/lib/maas/rsyslog",
                 "/var/log/apache2*",
                 "/var/log/maas*",
                 "/var/log/upstart/maas-*",


Test patch:
Repository: sosreport/sos

Issue:
python 3.12: sos crashes because of `ConfigParser`
Iniitially reported as [RHBZ#2223526](https://bugzilla.redhat.com/show_bug.cgi?id=2223526)

`sos report` crashes with: `Could not initialize 'report': 'ConfigParser' object has no attribute 'readfp'`


Discussion:

Resolving patch:
diff --git a/sos/options.py b/sos/options.py
--- a/sos/options.py
+++ b/sos/options.py
@@ -226,7 +226,7 @@ def _update_from_section(section, config):
         try:
             try:
                 with open(config_file) as f:
-                    config.readfp(f)
+                    config.read_file(f, config_file)
             except DuplicateOptionError as err:
                 raise exit("Duplicate option '%s' in section '%s' in file %s"
                            % (err.option, err.section, config_file))


Test patch: