internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: zalando/patroni

Issue:
features/steps/standby_cluster.py produces error under Windows
Trying to execute:
`>behave -i standby_cluster.feature`
will produce error:
```
2020-05-07 19:27:19,407 ERROR: Failed to execute ['c:userspashaappdatalocalprogramspythonpython37-32python.exe', 'features/callback.py', '5362', 'on_role_change', 'master', 'batman']
Traceback (most recent call last):
  File "C:\Users\pasha\Code\zalando-patroni\patroni\postgresql\cancellable.py", line 28, in _start_process
    self._process = psutil.Popen(cmd, *args, **kwargs)
  File "c:\users\pasha\appdata\local\programs\python\python37-32\lib\site-packages\psutil\__init__.py", line 1431, in __init__
    self.__subproc = subprocess.Popen(*args, **kwargs)
  File "c:\users\pasha\appdata\local\programs\python\python37-32\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "c:\users\pasha\appdata\local\programs\python\python37-32\lib\subprocess.py", line 1178, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
```

This error produced by `postgresql/cancellable.py`: 
```
    def _start_process(self, cmd, *args, **kwargs):
        """This method must be executed only when the `_lock` is acquired"""

        try:
            self._process_children = []
            self._process_cmd = cmd
            self._process = psutil.Popen(cmd, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^
        except Exception:
            return logger.exception('Failed to execute %s',  cmd)
        return True
```



Discussion:

Resolving patch:
diff --git a/features/steps/standby_cluster.py b/features/steps/standby_cluster.py
--- a/features/steps/standby_cluster.py
+++ b/features/steps/standby_cluster.py
@@ -10,7 +10,8 @@
 WHERE application_name = '{0}'
 """
 
-callback = sys.executable + " features/callback2.py "
+executable = sys.executable if os.name != 'nt' else sys.executable.replace('\\', '/')
+callback = executable + " features/callback2.py "
 
 
 @step('I start {name:w} with callback configured')
@@ -18,7 +19,7 @@ def start_patroni_with_callbacks(context, name):
     return context.pctl.start(name, custom_config={
         "postgresql": {
             "callbacks": {
-                "on_role_change": sys.executable + " features/callback.py"
+                "on_role_change": executable + " features/callback.py"
             }
         }
     })
@@ -31,7 +32,7 @@ def start_patroni(context, name, cluster_name):
         "postgresql": {
             "callbacks": {c: callback + name for c in ('on_start', 'on_stop', 'on_restart', 'on_role_change')},
             "backup_restore": {
-                "command": (sys.executable + " features/backup_restore.py --sourcedir=" +
+                "command": (executable + " features/backup_restore.py --sourcedir=" +
                             os.path.join(context.pctl.patroni_path, 'data', 'basebackup'))}
         }
     })


Test patch:
Repository: zalando/patroni

Issue:
Secured connection to zookeeper
It looks like that for now Patroni doesn't support security configuration for Patroni-Zookeeper connectivity (TLS, ceritificates) in the same way like it is done for etcd.
Is it planned to implement Zookeeper secured communication in the nearest future?


Discussion:
Hi @oleg-beltz,

we don't have such plans.
This is an open-source project, if you are interested in a specific feature, feel free to work on it. All contributions are always welcomed.

Resolving patch:
diff --git a/patroni/config.py b/patroni/config.py
--- a/patroni/config.py
+++ b/patroni/config.py
@@ -307,7 +307,7 @@ def _parse_list(value):
                               'CACERT', 'CERT', 'KEY', 'VERIFY', 'TOKEN', 'CHECKS', 'DC', 'CONSISTENCY',
                               'REGISTER_SERVICE', 'SERVICE_CHECK_INTERVAL', 'NAMESPACE', 'CONTEXT',
                               'USE_ENDPOINTS', 'SCOPE_LABEL', 'ROLE_LABEL', 'POD_IP', 'PORTS', 'LABELS',
-                              'BYPASS_API_SERVICE') and name:
+                              'BYPASS_API_SERVICE', 'KEY_PASSWORD', 'USE_SSL') and name:
                     value = os.environ.pop(param)
                     if suffix == 'PORT':
                         value = value and parse_int(value)
diff --git a/patroni/dcs/zookeeper.py b/patroni/dcs/zookeeper.py
--- a/patroni/dcs/zookeeper.py
+++ b/patroni/dcs/zookeeper.py
@@ -63,10 +63,14 @@ def __init__(self, config):
         if isinstance(hosts, list):
             hosts = ','.join(hosts)
 
+        mapping = {'use_ssl': 'use_ssl', 'verify': 'verify_certs', 'cacert': 'ca',
+                   'cert': 'certfile', 'key': 'keyfile', 'key_password': 'keyfile_password'}
+        kwargs = {v: config[k] for k, v in mapping.items() if k in config}
+
         self._client = KazooClient(hosts, handler=PatroniSequentialThreadingHandler(config['retry_timeout']),
                                    timeout=config['ttl'], connection_retry=KazooRetry(max_delay=1, max_tries=-1,
                                    sleep_func=time.sleep), command_retry=KazooRetry(deadline=config['retry_timeout'],
-                                   max_delay=1, max_tries=-1, sleep_func=time.sleep))
+                                   max_delay=1, max_tries=-1, sleep_func=time.sleep), **kwargs)
         self._client.add_listener(self.session_listener)
 
         self._fetch_cluster = True


Test patch:
Repository: zalando/patroni

Issue:
no_params for custom bootstrap method
**Please** add support the `no_params` parameter for the custom cluster bootstrap method.

> A special no_params parameter, if defined, restricts passing parameters to custom command.

In order not to wrap the restore command to a script.

Example:
```
bootstrap:
  method: wal_g
  wal_g:
    command: wal-g backup-fetch {{ postgresql_data_dir }} LATEST
    no_params: True
    recovery_conf: 
       recovery_target_action: promote
       recovery_target_timeline: latest
       restore_command: wal-g wal-fetch %f %p
```



Discussion:
There are two options:
1. You implement the feature
2. You wrap wal-g into a shell script
Ok

Resolving patch:
diff --git a/patroni/postgresql/bootstrap.py b/patroni/postgresql/bootstrap.py
--- a/patroni/postgresql/bootstrap.py
+++ b/patroni/postgresql/bootstrap.py
@@ -98,7 +98,8 @@ def _post_restore(self):
 
     def _custom_bootstrap(self, config):
         self._postgresql.set_state('running custom bootstrap script')
-        params = ['--scope=' + self._postgresql.scope, '--datadir=' + self._postgresql.data_dir]
+        params = [] if config.get('no_params') else ['--scope=' + self._postgresql.scope,
+                                                     '--datadir=' + self._postgresql.data_dir]
         try:
             logger.info('Running custom bootstrap script: %s', config['command'])
             if self._postgresql.cancellable.call(shlex.split(config['command']) + params) != 0:


Test patch:
Repository: zalando/patroni

Issue:
libpq's sslpassword parameter support?
PostgreSQL 13 introduced a new connection parameter to unlock encrypted client certificate key files called [`sslpassword`](https://www.postgresql.org/docs/13/libpq-connect.html#LIBPQ-CONNECT-SSLPASSWORD). I'm not sure if it's excluded intentionally from all Postgres 13 related PRs? I would like to submit a PR on this if you don't have any objections.



Discussion:
Sure, please do.
You can take https://github.com/zalando/patroni/pull/1523/files as an example.

Resolving patch:
diff --git a/patroni/config.py b/patroni/config.py
--- a/patroni/config.py
+++ b/patroni/config.py
@@ -21,6 +21,7 @@
     'sslmode',
     'sslcert',
     'sslkey',
+    'sslpassword',
     'sslrootcert',
     'sslcrl',
     'gssencmode',
diff --git a/patroni/postgresql/config.py b/patroni/postgresql/config.py
--- a/patroni/postgresql/config.py
+++ b/patroni/postgresql/config.py
@@ -478,7 +478,7 @@ def primary_conninfo_params(self, member):
     def format_dsn(self, params, include_dbname=False):
         # A list of keywords that can be found in a conninfo string. Follows what is acceptable by libpq
         keywords = ('dbname', 'user', 'passfile' if params.get('passfile') else 'password', 'host', 'port',
-                    'sslmode', 'sslcompression', 'sslcert', 'sslkey', 'sslrootcert', 'sslcrl',
+                    'sslmode', 'sslcompression', 'sslcert', 'sslkey', 'sslpassword', 'sslrootcert', 'sslcrl',
                     'application_name', 'krbsrvname', 'gssencmode', 'channel_binding')
         if include_dbname:
             params = params.copy()


Test patch:
Repository: zalando/patroni

Issue:
Create raft data_dir if it does not exist
I think it would be useful to have patroni/patroni_raft_controller create the `data_dir` if it does not exist already. If the user that starts patroni/patroni_raft_controller does not have the rigts to create that directory, that would still be a failure.

This would make setting things up a bit easier.


Discussion:

Resolving patch:
diff --git a/patroni/dcs/raft.py b/patroni/dcs/raft.py
--- a/patroni/dcs/raft.py
+++ b/patroni/dcs/raft.py
@@ -5,6 +5,7 @@
 import time
 
 from patroni.dcs import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, SyncState, TimelineHistory
+from ..utils import validate_directory
 from pysyncobj import SyncObj, SyncObjConf, replicated, FAIL_REASON
 from pysyncobj.transport import Node, TCPTransport, CONNECTION_STATE
 
@@ -275,6 +276,11 @@ def __init__(self, config):
                 partner_addrs.append(self_addr)
             self_addr = None
 
+        # Create raft data_dir if necessary
+        raft_data_dir = config.get('data_dir', '')
+        if raft_data_dir != '':
+            validate_directory(raft_data_dir)
+
         ready_event = threading.Event()
         file_template = os.path.join(config.get('data_dir', ''), (self_addr or ''))
         conf = SyncObjConf(password=config.get('password'), appendEntriesUseBatch=False,
diff --git a/patroni/utils.py b/patroni/utils.py
--- a/patroni/utils.py
+++ b/patroni/utils.py
@@ -1,3 +1,4 @@
+import errno
 import json.decoder as json_decoder
 import logging
 import os
@@ -461,7 +462,8 @@ def validate_directory(d, msg="{} {}"):
             os.makedirs(d)
         except OSError as e:
             logger.error(e)
-            raise PatroniException(msg.format(d, "couldn't create the directory"))
+            if e.errno != errno.EEXIST:
+                raise PatroniException(msg.format(d, "couldn't create the directory"))
     elif os.path.isdir(d):
         try:
             fd, tmpfile = tempfile.mkstemp(dir=d)


Test patch:
Repository: zalando/patroni

Issue:
Disable creation of PGPASS if using kerberos
Hello,

It doesn't make sense to create `PGPASS` file if we are using [kerberos](https://github.com/zalando/patroni/pull/1015). Can we disable in that case?

PGPASS is being created in [here](https://github.com/zalando/patroni/blob/881bba9e1cd081f7cd4c340f9a6ad977daa92c58/patroni/postgresql/config.py#L369).

I know that we don't populate or use PGPASS when kerberos is enabled, but creating parent dir of default PGPASS file could be a trouble when the default(~) directory doesn't exist on a node.


Discussion:
@RajKiranS you are right. Feel free to create a PR implementing the necessary check.

Resolving patch:
diff --git a/patroni/postgresql/config.py b/patroni/postgresql/config.py
--- a/patroni/postgresql/config.py
+++ b/patroni/postgresql/config.py
@@ -336,8 +336,9 @@ def check_directories(self):
         if "stats_temp_directory" in self._server_parameters:
             self.try_to_create_dir(self._server_parameters["stats_temp_directory"],
                                    "'{}' is defined in stats_temp_directory, {}")
-        self.try_to_create_dir(os.path.dirname(self._pgpass),
-                               "'{}' is defined in `postgresql.pgpass`, {}")
+        if not self._krbsrvname:
+            self.try_to_create_dir(os.path.dirname(self._pgpass),
+                                   "'{}' is defined in `postgresql.pgpass`, {}")
 
     @property
     def _configuration_to_save(self):


Test patch:
Repository: zalando/patroni

Issue:
 minor typo in logging message
`promoted self to leader because i had the session lock` instead of `promoted self to leader because I had the session lock` (note the capitalization of `I`) in `patroni/patroni/ha.py` and `patroni/tests/test_ha.py`.


Discussion:

Resolving patch:
diff --git a/patroni/ha.py b/patroni/ha.py
--- a/patroni/ha.py
+++ b/patroni/ha.py
@@ -1002,7 +1002,7 @@ def process_healthy_cluster(self):
                 else:
                     return self.enforce_master_role(
                         'no action. I am ({0}) the leader with the lock'.format(self.state_handler.name),
-                        'promoted self to leader because i had the session lock'
+                        'promoted self to leader because I had the session lock'
                     )
             else:
                 # Either there is no connection to DCS or someone else acquired the lock


Test patch:
diff --git a/tests/test_ha.py b/tests/test_ha.py
--- a/tests/test_ha.py
+++ b/tests/test_ha.py
@@ -300,7 +300,7 @@ def test_start_as_readonly(self):
         self.p.is_healthy = true
         self.ha.has_lock = true
         self.p.controldata = lambda: {'Database cluster state': 'in production', 'Database system identifier': SYSID}
-        self.assertEqual(self.ha.run_cycle(), 'promoted self to leader because i had the session lock')
+        self.assertEqual(self.ha.run_cycle(), 'promoted self to leader because I had the session lock')
 
     @patch('psycopg2.connect', psycopg2_connect)
     def test_acquire_lock_as_master(self):
@@ -354,7 +354,7 @@ def test_promote_because_have_lock(self):
         self.ha.cluster.is_unlocked = false
         self.ha.has_lock = true
         self.p.is_leader = false
-        self.assertEqual(self.ha.run_cycle(), 'promoted self to leader because i had the session lock')
+        self.assertEqual(self.ha.run_cycle(), 'promoted self to leader because I had the session lock')
 
     def test_promote_without_watchdog(self):
         self.ha.cluster.is_unlocked = false
Repository: zalando/patroni

Issue:
documentation build failure with sphinx-4.2.0
**Describe the bug**

On Debian unstable, patroni fails to build with the latest sphinx:

```
Running Sphinx v4.2.0
making output directory... done

Exception occurred:
  File "/<<PKGBUILDDIR>>/docs/conf.py", line 207, in setup
    app.add_stylesheet("custom.css")
AttributeError: 'Sphinx' object has no attribute 'add_stylesheet'
```


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -194,4 +194,7 @@
 # A possibility to have an own stylesheet, to add new rules or override existing ones
 # For the latter case, the CSS specificity of the rules should be higher than the default ones
 def setup(app):
-   app.add_stylesheet("custom.css")
+    if hasattr(app, 'add_css_file'):
+        app.add_css_file('custom.css')
+    else:
+        app.add_stylesheet('custom.css')


Test patch:
Repository: zalando/patroni

Issue:
etcd machines_cache can have duplicate entries
**Describe the bug**

While researching retries in case of DCS failures, I noticed that `_calculate_timeouts()` gets 7 as `etcd_nodes` when I use etcd, but only 3 if I use etcd3. In both cases, etcd is running locally on a 3-node Patroni cluster, and all three nodes are mentioned in the `etcd:` (or `etcd3`) section.

Not sure where the difference is from, but `machines_cache()` for `etcd` gets three times three nodes, and removes only the duplicate `base_uri` (the etcd leader it seems) hosts, not the others, so 7 remain.

This implies that `retry_timeout` with  a 3-node etcd cluster is divided by 7 for `etcd` and only 3 for `etcd3`, which looks wrong to me? On the other hand, it is not a big deal.

**Expected behavior**

Same timeout for etcd and etcd3.

**Environment**
- Patroni version: 2.1.1
- PostgreSQL version: 14.1
- DCS (and its version): etcd-3.3.25

**Patroni configuration file**
```
etcd:
  hosts:
    - 10.0.3.195:2379
    - 10.0.3.237:2379
    - 10.0.3.178:2379
```


Discussion:
etcd.hosts is only the initial list, it is used to get the topology from etcd itself.
You need to check what cluster returns as advertiseClientURLs or something like this.

Resolving patch:
diff --git a/patroni/dcs/etcd.py b/patroni/dcs/etcd.py
--- a/patroni/dcs/etcd.py
+++ b/patroni/dcs/etcd.py
@@ -184,7 +184,7 @@ def machines(self):
 
         for base_uri in machines_cache:
             try:
-                machines = list(self._get_members(base_uri, **kwargs))
+                machines = list(set(self._get_members(base_uri, **kwargs)))
                 logger.debug("Retrieved list of machines: %s", machines)
                 if machines:
                     random.shuffle(machines)


Test patch:
Repository: zalando/patroni

Issue:
Rest API somtimes stops responding when switching certificates
Switching REST API certificates while there is a concurrent API request active results in the following error:

    2022-01-25 14:14:37,739 ERROR: Failed to reload config_file=postgres0.yml
    Traceback (most recent call last):
      File "/home/ants/work/patroni/patroni/patroni/__main__.py", line 74, in reload_config
        self.api.reload_config(self.config['restapi'])
      File "/home/ants/work/patroni/patroni/patroni/api.py", line 871, in reload_config
        self.__initialize(config['listen'], ssl_options)
      File "/home/ants/work/patroni/patroni/patroni/api.py", line 776, in __initialize
        self.__httpserver_init(host, port)
      File "/home/ants/work/patroni/patroni/patroni/api.py", line 755, in __httpserver_init
        HTTPServer.__init__(self, info[0][-1][:2], RestApiHandler)
      File "/usr/lib/python3.8/socketserver.py", line 452, in __init__
        self.server_bind()
      File "/usr/lib/python3.8/http/server.py", line 138, in server_bind
        socketserver.TCPServer.server_bind(self)
      File "/usr/lib/python3.8/socketserver.py", line 466, in server_bind
        self.socket.bind(self.server_address)
    OSError: [Errno 98] Address already in use

After this REST API is stuck in a state where it is not responding.

To reproduce:

* Create two keypairs {cert,key}{1,2}.pem, one with EE as country other with AU
* Configure postgres0.yml to use the keypair.
* Run attached python script to create a load on the API.
* Run attached shell script to swap between the certs continuously. Should get stuck within a few tries.


Discussion:

Resolving patch:
diff --git a/patroni/api.py b/patroni/api.py
--- a/patroni/api.py
+++ b/patroni/api.py
@@ -768,6 +768,8 @@ def __initialize(self, listen, ssl_options):
         reloading_config = self.__listen is not None  # changing config in runtime
         if reloading_config:
             self.shutdown()
+            # Rely on ThreadingMixIn.server_close() to have all requests terminate before we continue
+            self.server_close()
 
         self.__listen = listen
         self.__ssl_options = ssl_options


Test patch:
diff --git a/tests/test_api.py b/tests/test_api.py
--- a/tests/test_api.py
+++ b/tests/test_api.py
@@ -549,7 +549,8 @@ def test_reload_config(self):
         self.assertRaises(ValueError, MockRestApiServer, None, '', bad_config)
         self.assertRaises(ValueError, self.srv.reload_config, bad_config)
         self.assertRaises(ValueError, self.srv.reload_config, {})
-        with patch.object(socket.socket, 'setsockopt', Mock(side_effect=socket.error)):
+        with patch.object(socket.socket, 'setsockopt', Mock(side_effect=socket.error)), \
+                patch.object(MockRestApiServer, 'server_close', Mock()):
             self.srv.reload_config({'listen': ':8008'})
 
     @patch.object(MockPatroni, 'dcs')
Repository: google-research/text-to-text-transfer-transformer

Issue:
Do we support GPU distributed training?
Hi, thanks for the awesome project!

Does the code base support distributed training? If not, is it possible to support it after some code modifications?

By the way, what is the way to set batch size and gpu number if I want to use GPU to train the model?

Thank you for your kind attention.


Discussion:
Hey Morizeyao, thanks for your interest. 

> Does the code base support distributed training? If not, is it possible to support it after some code modifications?

We (the authors) haven't tried GPU training yet but it should be possible. See the example under here: https://github.com/tensorflow/mesh#example-network-mnist
@nshazeer in case he wants to chime in.

> By the way, what is the way to set batch size and gpu number if I want to use GPU to train the model?

The batch size can be set according to how much memory your GPU can hold. The Mesh TF Transformer is in principle smart enough to chop your batch up into microbatches and accumulate gradients if the batch won't fit in memory. The GPU number will depend on which GPUs are available on the machine you're running on and which you want to use. Or do you mean how many GPUs you should use? It should work to use as many as you have available.
Thank you for your quick response!

About distributed GPU training, I'll do some research following your link. :)

For the second question, I did some test on 2 GPU machine and the code used only one card. 
As you have said, the Mesh library automatically choose the batch size, gradient accumulation and gpu numbers. I guess the problem may due to the environment settings.

I will report my findings here soon.
Same question here. I have tried my best to let the code run on GPUs, but the log shows that tpu_estimator.py says training on CPU. And the speed is around 300 times slower than on tpu in GCP. I believe I have set settings properly. The log_device_placement show that some of the operations are assigned to cpu, and some assigned to gpu. Do you have any idea what is going on?
> And the speed is around 300 times slower than on tpu in GCP.  

How did you get to this number? 

Have you tried changing your batch size? 
As @craffel mentioned: 
> The batch size can be set according to how much memory your GPU can hold.  
Verified a fix. Working on a PR.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -22,7 +22,7 @@
 
 setuptools.setup(
     name='t5',
-    version='0.1.7',
+    version='0.1.8',
     description='Text-to-text transfer transformer',
     long_description=_LONG_DESCRIPTION,
     long_description_content_type='text/markdown',
@@ -41,7 +41,7 @@
         'babel',
         'future',
         'gin-config',
-        'mesh-tensorflow[transformer]>=0.1.8',
+        'mesh-tensorflow[transformer]>=0.1.9',
         'nltk',
         'numpy',
         'pandas',


Test patch: