internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: weecology/retriever

Issue:
Download from Kaggle function needs to be updated
The function [download_from_kaggle](https://github.com/weecology/retriever/blob/fe5148b66a44fd94ba714a5cf8f9cda4f35e3249/retriever/lib/engine.py#L512) uses the Kaggle API to download datasets. This functions need to be updated to work well.


Discussion:

Resolving patch:
diff --git a/retriever/lib/engine.py b/retriever/lib/engine.py
--- a/retriever/lib/engine.py
+++ b/retriever/lib/engine.py
@@ -533,34 +533,30 @@ def download_from_kaggle(
 
         api = KaggleApi()
         api.authenticate()
+        archive_full_path = archive_full_path + ".zip"
 
         if data_source == "dataset":
-            archive_full_path = archive_full_path + ".zip"
             try:
                 api.dataset_download_files(dataset=dataset_name,
                                            path=archive_dir,
                                            quiet=False,
                                            force=True)
                 file_names = self.extract_zip(archive_full_path, archive_dir)
-            except ApiException:
-                print(f"The dataset '{dataset_name}' isn't currently available "
-                      f"in the Retriever.\nRun 'retriever ls' to see a "
-                      f"list of currently available datasets.")
+            except ApiException as e:
+                print(f"Could not download '{dataset_name}' from kaggle datasets.")
+                print("Response :", e.body.decode("unicode_escape"))
                 return []
 
         else:
-            archive_full_path = archive_full_path.replace("kaggle:competition:",
-                                                          "") + ".zip"
             try:
                 api.competition_download_files(competition=dataset_name,
                                                path=archive_dir,
                                                quiet=False,
                                                force=True)
                 file_names = self.extract_zip(archive_full_path, archive_dir)
-            except ApiException:
-                print(f"The dataset '{dataset_name}' isn't currently available "
-                      f"in the Retriever.\nRun 'retriever ls' to see a "
-                      f"list of currently available datasets.")
+            except ApiException as e:
+                print(f"Could not download '{dataset_name}' from kaggle competitions.")
+                print("Response :", e.body.decode("unicode_escape"))
                 return []
 
         return file_names
@@ -589,8 +585,7 @@ def download_files_from_archive(
             archive_dir = archive_dir.format(dataset=self.script.name)
             if not os.path.exists(archive_dir):
                 os.makedirs(archive_dir)
-
-        if hasattr(self.script.__dict__, "kaggle"):
+        if hasattr(self.script, "kaggle"):
             file_names = self.download_from_kaggle(data_source=self.script.data_source,
                                                    dataset_name=url,
                                                    archive_dir=archive_dir,


Test patch:
Repository: weecology/retriever

Issue:
Clean up CLI to use the python interface

In the python interface, we use the function [def datasets(keywords=None, licenses=None):](https://github.com/weecology/retriever/blob/bb06180a030d34eafa2e1ea13b74a3719df827e1/retriever/lib/datasets.py#L5).
This should be able to return what we are doing in the main (CLI interface) at [line](https://github.com/weecology/retriever/blob/main/retriever/__main__.py#L105).
 and the results can be printed in the away that looks good in CLI(terminal)

_Originally posted by @henrykironde in https://github.com/weecology/retriever/issues/1570#issuecomment-810558636_


Discussion:
@Aakash3101 
Okay, so from what I understood is that the ```retriever ls -v``` command should also display the error message that at least one argument should be passed, right?
Yes
I found this in ```lib/get_opts.py```

```
ls_parser.add_argument('-v',
                       help='verbose list of all datasets',
                       nargs='*',
                       default=False)
```
I think it was intended to display the verbose list of all datasets when no arguments are provided.

Also when I run ```retriever ls -v airports``` it displays the verbose for the airports dataset.
```
aakash01@aakash01-G3-3579:~/Desktop/gsoc/retriever (verbose-error)$ retriever ls -v airports
1. Dataset containing information on all airports on ouraiports.com
 airports
['airports', 'aeroplanes', 'airplanes', 'frequencies', 'runways', 'navaids', 'countries', 'regions']
Dataset containing information on all airports on ourairports.com
None
OurAirports.com, Megginson Technologies Ltd.
```
I just want to clarify before making any changes

Resolving patch:
diff --git a/retriever/lib/get_opts.py b/retriever/lib/get_opts.py
--- a/retriever/lib/get_opts.py
+++ b/retriever/lib/get_opts.py
@@ -31,9 +31,10 @@
 script_list.extend(get_dataset_names_upstream())
 script_list = sorted(set(script_list))
 
-# set of all possible licenses and keywords
+# set of all possible licenses, keywords and scripts
 licenses_options = set(licenses_list)
 keywords_options = set(keywords_list)
+scripts_options = script_list
 
 parser = argparse.ArgumentParser(prog="retriever")
 parser.add_argument('-v', '--version', action='version', version=VERSION)
@@ -121,10 +122,8 @@
                        nargs='+').completer = ChoicesCompleter(list(licenses_options))
 ls_parser.add_argument('-k', help='search datasets with keyword(s)',
                        nargs='+').completer = ChoicesCompleter(list(keywords_options))
-ls_parser.add_argument('-v',
-                       help='verbose list of all datasets',
-                       nargs='*',
-                       default=False)
+ls_parser.add_argument('-v', help='verbose list of specified dataset(s)',
+                       nargs='+').completer = ChoicesCompleter(list(scripts_options))
 
 autocreate_parser.add_argument('path', help='path to the data file(s)')
 autocreate_parser.add_argument('-dt',


Test patch:
Repository: weecology/retriever

Issue:
`retriever download` command throwing error.
After running `retriever download iris`, I am getting this error : 
![image](https://files.gitter.im/606462c26da037398478bdfb/LB3L/image.png)


Discussion:
I can confirm this as a general error with `download` from the CLI in the current `main`. The issue is not present in the current release and is also does not appear to be present in the Python interface for current `main` (which is why the tests aren't catching it).

```python
In [11]: rt.download('iris')
=> Downloading iris
Copying bezdekIris.data from /home/ethan/.retriever/raw_data/iris
Out[11]: <retriever.engines.download_only.engine at 0x7fd615759790>
```

Resolving patch:
diff --git a/retriever/lib/install.py b/retriever/lib/install.py
--- a/retriever/lib/install.py
+++ b/retriever/lib/install.py
@@ -13,9 +13,9 @@ def _install(args, use_cache, debug):
     engine = choose_engine(args)
     engine.use_cache = use_cache
 
-    if args['dataset'].endswith('.zip') or args['hash_value']:
+    if args['dataset'].endswith('.zip') or args.get('hash_value'):
         path_to_archive = args['dataset']
-        if args['hash_value']:
+        if args.get('hash_value'):
             path_to_archive = os.path.join(
                 PROVENANCE_DIR, args['dataset'],
                 '{}-{}.zip'.format(args['dataset'], args['hash_value']))


Test patch:
Repository: zalando/patroni

Issue:
AttributeError: 'KazooClient' object has no attribute '_retry'
Using `patronictl` to pause the cluster management, a configuration reload is triggered on nodes leading to the following exception with ZooKeeper:
```
2017-07-02 16:31:14,690 ERROR: Failed to reload config_file=/etc/patroni/node03.yml
Traceback (most recent call last):
  File "/opt/patroni/patroni/__init__.py", line 67, in reload_config
    self.dcs.reload_config(self.config)
  File "/opt/patroni/patroni/dcs/zookeeper.py", line 92, in reload_config
    self.set_retry_timeout(config['retry_timeout'])
  File "/opt/patroni/patroni/dcs/zookeeper.py", line 119, in set_retry_timeout
    self._client._retry.deadline = retry_timeout
AttributeError: 'KazooClient' object has no attribute '_retry'
```
Running on Debian 8 with Python 2.7.9 and Kazoo 1.3.1.


Discussion:

Resolving patch:
diff --git a/patroni/dcs/zookeeper.py b/patroni/dcs/zookeeper.py
--- a/patroni/dcs/zookeeper.py
+++ b/patroni/dcs/zookeeper.py
@@ -116,7 +116,8 @@ def set_ttl(self, ttl):
             return True
 
     def set_retry_timeout(self, retry_timeout):
-        self._client._retry.deadline = retry_timeout
+        retry = self._client.retry if isinstance(self._client.retry, KazooRetry) else self._client._retry
+        retry.deadline = retry_timeout
 
     def get_node(self, key, watch=None):
         try:


Test patch:
Repository: zalando/patroni

Issue:
Issues when using custom bootstrap
When trying to perform a restore using wal-e, on a Spilo image, Spilo tries to modify the template1 database. 
Patroni creates a temprary pg_hba file for use during the custom bootstrap, but it only allows to connect to the "postgres" db ([line 1140 or postgresql.py](https://github.com/zalando/patroni/blame/master/patroni/postgresql.py#L1140)).
In order to allow Spilo to create those extensions, instead of setting up "postgres" as the db, it would be handy to set "all" there.

As far as I could see there is no way to pass that as parameter, am I right?

Ideas?
Thanks


Discussion:
Hi @francobellagamba,

you are right, my level of paranoia was way to high when I did that and access should be open for "all".
We need to fix that and this is a nice opportunity to create a simple pull request :)

Actually I have a question in return.
Have you modified Spilo image to perform custom bootstrap? Can we look on your changes?
Was the backup made from another Spilo cluster? If yes, than it's not really necessary to run post_bootstrap script, all the stuff is already there and Patroni only wants to change superuser and replication passwords.

Resolving patch:
diff --git a/patroni/postgresql.py b/patroni/postgresql.py
--- a/patroni/postgresql.py
+++ b/patroni/postgresql.py
@@ -1137,7 +1137,7 @@ def _replace_pg_hba(self):
             with open(self._pg_hba_conf, 'w') as f:
                 f.write(self._CONFIG_WARNING_HEADER)
                 for address, t in addresses.items():
-                    f.write('{0}\t{1}\t{2}\t{3}\ttrust\n'.format(t, self._database,
+                    f.write('{0}\t{1}\t{2}\t{3}\ttrust\n'.format(t, 'all',
                                                                  self._superuser.get('username') or 'all', address))
         elif not self._server_parameters.get('hba_file') and self.config.get('pg_hba'):
             with open(self._pg_hba_conf, 'w') as f:


Test patch:
Repository: zalando/patroni

Issue:
Add new healthcheck endpoint to Patroni api
It should return 200 only if the node is running as a synchronous replica (when it's exposed as synchronous in `sync` key in the DCS).


Discussion:

Resolving patch:
diff --git a/patroni/api.py b/patroni/api.py
--- a/patroni/api.py
+++ b/patroni/api.py
@@ -80,6 +80,14 @@ def do_GET(self, write_status_code_only=False):
 
         patroni = self.server.patroni
         cluster = patroni.dcs.cluster
+
+        def is_synchronous():
+            return (cluster.is_synchronous_mode() and cluster.sync
+                    and cluster.sync.sync_standby == patroni.postgresql.name)
+
+        def is_balanceable_replica():
+            return response.get('role') == 'replica' and not patroni.noloadbalance
+
         if cluster:  # dcs available
             if cluster.leader and cluster.leader.name == patroni.postgresql.name:  # is_leader
                 status_code = 200 if 'master' in path else 503
@@ -87,6 +95,10 @@ def do_GET(self, write_status_code_only=False):
                 status_code = 503
             elif response['role'] == 'master':  # running as master but without leader lock!!!!
                 status_code = 503
+            elif path in ('/sync', '/synchronous'):
+                status_code = 200 if is_balanceable_replica() and is_synchronous() else 503
+            elif path in ('/async', '/asynchronous'):
+                status_code = 200 if is_balanceable_replica() and not is_synchronous() else 503
             elif response['role'] in path:  # response['role'] != 'master'
                 status_code = 503 if patroni.noloadbalance else 200
             else:


Test patch:
diff --git a/tests/test_api.py b/tests/test_api.py
--- a/tests/test_api.py
+++ b/tests/test_api.py
@@ -92,7 +92,7 @@ class MockPatroni(object):
     dcs = Mock()
     tags = {}
     version = '0.00'
-    noloadbalance = Mock(return_value=False)
+    noloadbalance = PropertyMock(return_value=False)
     scheduled_restart = {'schedule': future_restart_time,
                          'postmaster_start_time': postgresql.postmaster_start_time()}
 
@@ -140,6 +140,12 @@ def test_do_GET(self):
         with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'master'})):
             MockRestApiServer(RestApiHandler, 'GET /replica')
         MockRestApiServer(RestApiHandler, 'GET /master')
+        MockPatroni.dcs.cluster.sync.sync_standby = MockPostgresql.name
+        MockPatroni.dcs.cluster.is_synchronous_mode = Mock(return_value=True)
+        with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'replica'})):
+            MockRestApiServer(RestApiHandler, 'GET /synchronous')
+        with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'replica'})):
+            MockRestApiServer(RestApiHandler, 'GET /asynchronous')
         MockPatroni.dcs.cluster.leader.name = MockPostgresql.name
         MockRestApiServer(RestApiHandler, 'GET /replica')
         MockPatroni.dcs.cluster = None
Repository: zalando/patroni

Issue:
Could not load library ".../libpqwalreceiver.so"
We compile and install Postgres in a non-standard location (/opt/custom/share/postgresql-9.6.2_7 in the example below) which requires that we specify the `LD_LIBRARY_PATH` env var when starting up postgres and when working with postgres utilities. When starting up Postgres, Patroni currently passes along `PATH`, `LC_ALL` and `LANG` env vars if they are set, but does not do the same with `LD_LIBRARY_PATH`. This results in the following FATAL replication error on the replica node:
```
2018-06-05 21:10:26,675 INFO: Lock owner: postgresdb-01; I am postgresdb-02
2018-06-05 21:10:26,675 INFO: does not have lock
2018-06-05 21:10:26,676 INFO: no action.  i am a secondary and i am following a leader
FATAL:  could not load library "/opt/custom/share/postgresql-9.6.2_7/lib/libpqwalreceiver.so": libpq.so.5: cannot open shared object file: No such file or directory
FATAL:  could not load library "/opt/custom/share/postgresql-9.6.2_7/lib/libpqwalreceiver.so": libpq.so.5: cannot open shared object file: No such file or directory
```

The fix for this is to pass `LD_LIBRARY_PATH` along with the other env vars, which results in replication behaving as expected:
```
2018-06-05 22:03:08,722 INFO: Lock owner: postgresdb-01; I am postgresdb-02
2018-06-05 22:03:08,722 INFO: does not have lock
2018-06-05 22:03:08,727 DEBUG: Sending request(xid=49): SetData(path='/service/postgres_cluster/members/postgresdb-02', data='{"conn_url":"postgres://postgresdb-02:5432/postgres","api_url":"http://postgresdb-02:8008/patroni","timeline":1,"state":"running","role":"replica","xlog_location":184664496}', version=-1)
2018-06-05 22:03:08,737 DEBUG: Received response(xid=49): ZnodeStat(czxid=4294967413, mzxid=4294967497, ctime=1528235848173, mtime=1528236188727, version=28, cversion=0, aversion=0, ephemeralOwner=216194533074731009, dataLength=169, numChildren=0, pzxid=4294967413)
2018-06-05 22:03:08,741 INFO: no action.  i am a secondary and i am following a leader
```

I will submit a pull request shortly that addresses this problem.


Discussion:

Resolving patch:
diff --git a/patroni/postmaster.py b/patroni/postmaster.py
--- a/patroni/postmaster.py
+++ b/patroni/postmaster.py
@@ -137,7 +137,7 @@ def start(pgcommand, data_dir, conf, options):
         # of init process to take care about postmaster.
         # In order to make everything portable we can't use fork&exec approach here, so  we will call
         # ourselves and pass list of arguments which must be used to start postgres.
-        env = {p: os.environ[p] for p in ('PATH', 'LC_ALL', 'LANG') if p in os.environ}
+        env = {p: os.environ[p] for p in ('PATH', 'LD_LIBRARY_PATH', 'LC_ALL', 'LANG') if p in os.environ}
         try:
             proc = PostmasterProcess._from_pidfile(data_dir)
             if proc and not proc._is_postmaster_process():


Test patch:
Repository: zalando/patroni

Issue:
Reasons of minimum PostgreSQL parameters value (max_connections, ...)
In `postresql/config.py` there is:

```
        'max_connections': (100, lambda v: int(v) >= 100, 90100),
        'max_wal_senders': (10, lambda v: int(v) >= 10, 90100),
        'wal_keep_segments': (8, lambda v: int(v) >= 8, 90100),
        # [...]
```

Is there any reason to have such "high" minimum in the validation method?

We're running small databases for customers where the number of connections for the first plan is 30, and we can't apply such configuration using patroni. (Same for `wal_keep_segments` and `max_wal_senders` which would be running correctly with lower values.

What about keeping the defaults and updating the minimum values?

ie

```
        'max_connections': (100, lambda v: int(v) >= 25, 90100),
        'max_wal_senders': (10, lambda v: int(v) >= 3, 90100),
        'wal_keep_segments': (8, lambda v: int(v) >= 1, 90100),
```

Regards,


Discussion:
Well, it is arguable if these values are "high", they actually mostly match current default values in the new cluster, but yes, we can make the minimal values smaller.
You can open a pull request if you like.
Thanks a lot for your response @CyberDem0n I will open a PR, I wanted to be sure that there wasn't a good reason I would have missed.

It's definitely arguable if these values are "high", but it is a good thing I think to let users use a lower one.

Resolving patch:
diff --git a/patroni/postgresql/config.py b/patroni/postgresql/config.py
--- a/patroni/postgresql/config.py
+++ b/patroni/postgresql/config.py
@@ -176,14 +176,14 @@ class ConfigHandler(object):
         'cluster_name': (None, lambda _: False, 90500),
         'wal_level': ('hot_standby', lambda v: v.lower() in ('hot_standby', 'replica', 'logical'), 90100),
         'hot_standby': ('on', lambda _: False, 90100),
-        'max_connections': (100, lambda v: int(v) >= 100, 90100),
-        'max_wal_senders': (10, lambda v: int(v) >= 10, 90100),
-        'wal_keep_segments': (8, lambda v: int(v) >= 8, 90100),
+        'max_connections': (100, lambda v: int(v) >= 25, 90100),
+        'max_wal_senders': (10, lambda v: int(v) >= 3, 90100),
+        'wal_keep_segments': (8, lambda v: int(v) >= 1, 90100),
         'max_prepared_transactions': (0, lambda v: int(v) >= 0, 90100),
-        'max_locks_per_transaction': (64, lambda v: int(v) >= 64, 90100),
+        'max_locks_per_transaction': (64, lambda v: int(v) >= 32, 90100),
         'track_commit_timestamp': ('off', lambda v: parse_bool(v) is not None, 90500),
-        'max_replication_slots': (10, lambda v: int(v) >= 10, 90400),
-        'max_worker_processes': (8, lambda v: int(v) >= 8, 90400),
+        'max_replication_slots': (10, lambda v: int(v) >= 4, 90400),
+        'max_worker_processes': (8, lambda v: int(v) >= 2, 90400),
         'wal_log_hints': ('on', lambda _: False, 90400)
     })
 


Test patch:
Repository: zalando/patroni

Issue:
`dest` argument default value is platform dependant
We should use `os.path.join` to handle path separators


Discussion:

Resolving patch:
diff --git a/features/environment.py b/features/environment.py
--- a/features/environment.py
+++ b/features/environment.py
@@ -258,7 +258,7 @@ def cancel_background(self):
     def backup_source(self):
         return 'postgres://{username}:{password}@{host}:{port}/{database}'.format(**self._replication)
 
-    def backup(self, dest='data/basebackup'):
+    def backup(self, dest=os.path.join('data', 'basebackup')):
         subprocess.call([PatroniPoolController.BACKUP_SCRIPT, '--walmethod=none',
                          '--datadir=' + os.path.join(self._work_directory, dest),
                          '--dbname=' + self.backup_source])


Test patch:
Repository: zalando/patroni

Issue:
Show tags in patronictl list -e
It would be nice to see tags when using `patronictl list`, at least with the `-e` option.  Currently we have to look at the .yml file.


Discussion:

Resolving patch:
diff --git a/patroni/ctl.py b/patroni/ctl.py
--- a/patroni/ctl.py
+++ b/patroni/ctl.py
@@ -726,7 +726,7 @@ def output_members(cluster, name, extended=False, fmt='pretty'):
     cluster = cluster_as_json(cluster)
 
     columns = ['Cluster', 'Member', 'Host', 'Role', 'State', 'TL', 'Lag in MB']
-    for c in ('Pending restart', 'Scheduled restart'):
+    for c in ('Pending restart', 'Scheduled restart', 'Tags'):
         if extended or any(m.get(c.lower().replace(' ', '_')) for m in cluster['members']):
             columns.append(c)
 
@@ -741,7 +741,8 @@ def output_members(cluster, name, extended=False, fmt='pretty'):
         m.update(cluster=name, member=m['name'], tl=m.get('timeline', ''),
                  role='' if m['role'] == 'replica' else m['role'].replace('_', ' ').title(),
                  lag_in_mb=round(lag/1024/1024) if isinstance(lag, six.integer_types) else lag,
-                 pending_restart='*' if m.get('pending_restart') else '')
+                 pending_restart='*' if m.get('pending_restart') else '',
+                 tags=json.dumps(m['tags']) if m.get('tags') else '')
 
         if append_port:
             m['host'] = ':'.join([m['host'], str(m['port'])])
@@ -754,7 +755,7 @@ def output_members(cluster, name, extended=False, fmt='pretty'):
 
         rows.append([m.get(n.lower().replace(' ', '_'), '') for n in columns])
 
-    print_output(columns, rows, {'Lag in MB': 'r', 'TL': 'r'}, fmt)
+    print_output(columns, rows, {'Lag in MB': 'r', 'TL': 'r', 'Tags': 'l'}, fmt)
 
     if fmt != 'pretty':  # Omit service info when using machine-readable formats
         return


Test patch: