Repository: patroni/patroni

Issue:
Latest ydiff version incompatibility
### What happened?

We just installed Patroni 4.0.3 and edit-config does not work because a compatibility issue with the latest version of ydiff.

Downgrading `ydiff` to 1.2.0 solves the issue.

```bash
[postgres@production-db-3 ~]$ patronictl -k edit-config
Traceback (most recent call last):
  File "/bin/patronictl", line 8, in <module>
    sys.exit(ctl())
             ^^^^^
  File "/applic/patroni_venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/applic/patroni_venv/lib64/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/applic/patroni_venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/applic/patroni_venv/lib64/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/applic/patroni_venv/lib64/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/applic/patroni_venv/lib64/python3.11/site-packages/patroni/ctl.py", line 2122, in edit_config
    show_diff(before_editing, after_editing)
  File "/applic/patroni_venv/lib64/python3.11/site-packages/patroni/ctl.py", line 1917, in show_diff
    markup_to_pager(PatchStream(buf), opts)
  File "/applic/patroni_venv/lib64/python3.11/site-packages/ydiff.py", line 640, in markup_to_pager
    theme=opts.theme)
          ^^^^^^^^^^
AttributeError: type object 'opts' has no attribute 'theme'
```

Library installed during the installation:
Package         Version
--------------- -----------
click           8.1.7
dnspython       2.7.0
patroni         4.0.3
pip             24.3.1
prettytable     3.12.0
psutil          6.1.0
psycopg2-binary 2.9.10
python-dateutil 2.9.0.post0
python-etcd     0.4.5
PyYAML          6.0.2
setuptools      75.5.0
six             1.16.0
urllib3         2.2.3
wcwidth         0.2.13
ydiff           1.4.1

### How can we reproduce it (as minimally and precisely as possible)?

Install latest patroni version and run
```bash
patronictl edit-config
```

### What did you expect to happen?

edit-config to work

### Patroni/PostgreSQL/DCS version

-   Patroni version: 4.0.3
-   PostgreSQL version: 13.13
-   DCS (and its version): ETCD 3.5.9


### Patroni configuration file

```yaml
scope: patroni
namespace: /service/
name: production-db-3

log:
  level: INFO
  dir: /var/log/patroni
  file_size: 50000000 # Size of log file which triggers a log rotation
  file_num: 10 # Number of logs to retain
  dateformat: '%Y-%m-%d %H:%M:%S'

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    master_start_timeout: 10
    synchronous_mode: true
    synchronous_mode_strict: false
    postgresql:
      use_pg_rewind: true
      use_slots: true
      keep_existing_recovery_conf: true
      recovery_conf:
        restore_command: /usr/bin/pgbackrest --stanza=production archive-get %f %p
        recovery_target_timeline: latest
  method: initdb
  initdb:
    - data-checksums
    - encoding: UTF8
    - locale: en_US.utf8
  pg_hba:
    - host all all 0.0.0.0/0 md5
    - host replication rep_user 0.0.0.0/0 md5

postgresql:
  listen: 0.0.0.0:5432
  connect_address: dsfdsfsdfdsfdsfsdfsdf.internal:5432
  data_dir: /data/postgres/13/pg_data

  bin_dir: /usr/pgsql-13/bin

  use_unix_socket: false
  pgpass: /home/postgres/.pgpass
  pg_ctl_timeout: 60
  use_pg_rewind: true
  remove_data_directory_on_rewind_failure: false
  authentication:
    superuser:
      username: postgres
    replication:
      username: repuser
    rewind:
      username: repuser
  create_replica_methods:
    - pgbackrest
  pgbackrest:
    command: /usr/bin/pgbackrest --log-level-file=detail --link-all --delta --stanza=production restore
    no_params: true
    keep_data: true

restapi:
  listen: 0.0.0.0:8008
  connect_address: production-db-3.internal:8008
  certfile: /etc/ssl/certs/production-db-3.c.internal.crt
  keyfile: /etc/ssl/certs/production-db-3.c.internal.key
  cafile: /etc/ssl/certs/production-db-3.internal.crt

watchdog:
  mode: automatic
  device: /dev/watchdog
  safety_margin: 5

etcd:
  host: 127.0.0.1:2379
  hosts:
    - 1.2.3.4:2379
    - 1.2.3.5:2379
    - 1.2.3.6:2379
  protocol: http

tags:
    nosync: true
    nofailover: true
```


### patronictl show-config

```yaml
loop_wait: 10
master_start_timeout: 10
maximum_lag_on_failover: 1048576
postgresql:
  keep_existing_recovery_conf: true
  parameters:
    max_locks_per_transaction: 512
  recovery_conf:
    recovery_target_timeline: latest
    restore_command: /usr/bin/pgbackrest --stanza=production archive-get %f %p
  use_pg_rewind: true
  use_slots: true
retry_timeout: 10
synchronous_mode: true
synchronous_mode_strict: false
```


### Patroni log files

```shell
N/A
```


### PostgreSQL log files

```shell
N/A
```


### Have you tried to use GitHub issue search?

- [X] Yes

### Anything else we need to know?

_No response_
Patroni should not use ydiff as a module
### What happened?

[patroni/ctl.py](https://github.com/patroni/patroni/blob/4ce0f99cfb5b08bf770a935c59936a6353e3d27d/patroni/ctl.py#L48) imports [ydiff](https://github.com/ymattw/ydiff) and calls their `markup_to_pager()` with a hacky [opts](https://github.com/patroni/patroni/blob/4ce0f99cfb5b08bf770a935c59936a6353e3d27d/patroni/ctl.py#L1897) to render colored diffs. The author of ydiff was not aware of this and changes they made often breaks patroni users, recent example is the new option `--theme` introduce (see https://github.com/patroni/patroni/issues/3212 and https://github.com/ymattw/ydiff/issues/124). 

Ydiff is intended to be used as a CLI tool, not a python module. Ideally a fix like https://github.com/patroni/patroni/pull/3214 should be applied, once we can make sure the environment Patroni runs always contain an executable `ydiff` or `ydiff.py` in `$PATH`.

### How can we reproduce it (as minimally and precisely as possible)?

See https://github.com/patroni/patroni/issues/3212.

### What did you expect to happen?

Patroni should not break after newer ydiff is released with new CLI options.

### Patroni/PostgreSQL/DCS version

N/a

### Patroni configuration file

```yaml
N/a
```


### patronictl show-config

```yaml
N/a
```


### Patroni log files

```shell
N/a
```


### PostgreSQL log files

```shell
N/a
```


### Have you tried to use GitHub issue search?

- [X] Yes

### Anything else we need to know?

_No response_


Discussion:

Patch:
diff --git a/patroni/ctl.py b/patroni/ctl.py
index d775c6ad6..2e485af2c 100644
--- a/patroni/ctl.py
+++ b/patroni/ctl.py
@@ -1889,12 +1889,13 @@ def listify(string: str) -> List[str]:
     unified_diff = difflib.unified_diff(listify(before_editing), listify(after_editing))
 
     if sys.stdout.isatty():
-        buf = io.StringIO()
+        buf = io.BytesIO()
         for line in unified_diff:
-            buf.write(str(line))
+            buf.write(line.encode('utf-8'))
         buf.seek(0)
 
         class opts:
+            theme = 'default'
             side_by_side = False
             width = 80
             tab_width = 8
diff --git a/requirements.txt b/requirements.txt
index a35d18655..5b3626b11 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -10,5 +10,5 @@ python-dateutil
 pysyncobj>=0.3.8
 cryptography>=1.4
 psutil>=2.0.0
-ydiff>=1.2.0
+ydiff>=1.2.0,<1.5,!=1.4.0,!=1.4.1
 python-json-logger>=2.0.2
diff --git a/typings/cdiff/__init__.pyi b/typings/cdiff/__init__.pyi
index 4578d4688..c23b6d733 100644
--- a/typings/cdiff/__init__.pyi
+++ b/typings/cdiff/__init__.pyi
@@ -1,5 +1,5 @@
 import io
 from typing import Any
 class PatchStream:
-    def __init__(self, diff_hdl: io.TextIOBase) -> None: ...
+    def __init__(self, diff_hdl: io.BytesIOBase) -> None: ...
 def markup_to_pager(stream: Any, opts: Any) -> None: ...
diff --git a/typings/ydiff/__init__.pyi b/typings/ydiff/__init__.pyi
index 4578d4688..c23b6d733 100644
--- a/typings/ydiff/__init__.pyi
+++ b/typings/ydiff/__init__.pyi
@@ -1,5 +1,5 @@
 import io
 from typing import Any
 class PatchStream:
-    def __init__(self, diff_hdl: io.TextIOBase) -> None: ...
+    def __init__(self, diff_hdl: io.BytesIOBase) -> None: ...
 def markup_to_pager(stream: Any, opts: Any) -> None: ...


Test patch:
diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
index 0e7709f24..f0f6d88c7 100644
--- a/.github/workflows/tests.yaml
+++ b/.github/workflows/tests.yaml
@@ -188,6 +188,23 @@ jobs:
       with:
         version: 1.1.385
 
+  ydiff:
+    name: Test compatibility with the latest version of ydiff
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set up Python 3.12
+      uses: actions/setup-python@v5
+      with:
+        python-version: 3.12
+    - name: Install dependencies
+      run: python .github/workflows/install_deps.py
+    - name: Update ydiff
+      run: python -m pip install -U ydiff
+    - name: Run tests
+      run: python -m pytest tests/test_ctl.py -v
+
   docs:
     runs-on: ubuntu-latest
     steps:
diff --git a/tests/test_ctl.py b/tests/test_ctl.py
index 07359ca6d..d137885be 100644
--- a/tests/test_ctl.py
+++ b/tests/test_ctl.py
@@ -682,6 +682,17 @@ def test_show_diff(self, mock_which, mock_env_get, mock_markup_to_pager, mock_is
         show_diff(b"foo:\n  bar: \xc3\xb6\xc3\xb6\n".decode('utf-8'),
                   b"foo:\n  bar: \xc3\xbc\xc3\xbc\n".decode('utf-8'))
 
+    @patch('subprocess.Popen')
+    @patch('os.environ.get', Mock(return_value='cat'))
+    @patch('sys.stdout.isatty', Mock(return_value=True))
+    @patch('shutil.which', Mock(return_value='cat'))
+    def test_show_diff_pager(self, mock_popen):
+        show_diff("foo:\n  bar: 1\n", "foo:\n  bar: 2\n")
+        self.assertEqual(mock_popen.return_value.stdin.write.call_count, 6)
+        self.assertIn(b' bar: ', mock_popen.return_value.stdin.write.call_args_list[5][0][0])
+        self.assertIn(b' bar: ', mock_popen.return_value.stdin.write.call_args_list[4][0][0])
+        self.assertIn(b' foo:', mock_popen.return_value.stdin.write.call_args_list[3][0][0])
+
     @patch('subprocess.call', return_value=1)
     def test_invoke_editor(self, mock_subprocess_call):
         os.environ.pop('EDITOR', None)