Repository: octodns/octodns-bind

Issue:
ZoneFileProvider writes IDN domains without encoding
I was aiming to create a scheduled backup of DNS zones hosted at one cloud provider in a bind compatible format using `octodns`, and to my surprise IDN records and zone file name were converted to their UTF-8 presentation when written out by `ZoneFileProvider`.

Is this intentional?

I have included a minimal reproducer:

```
$ cat idn-config.yml 
providers:
  zonefile-in:
    class: octodns_bind.ZoneFileProvider
    directory: ./input
    check_origin: false

  zonefile-out:
    class: octodns_bind.ZoneFileProvider
    directory: ./output

zones:
    '*':
        sources:
            - zonefile-in
        targets:
            - zonefile-out
$ cat input/xn--bcher-kva.example. 
$ORIGIN xn--bcher-kva.example.

@ 3600 IN SOA ns.xn--bcher-kva.example. hostmaster.xn--bcher-kva.example.(
    1732013604 ; Serial
    3600 ; Refresh
    600 ; Retry
    604800 ; Expire
    3600 ; NXDOMAIN ttl
)

@ 3600 IN A 127.0.0.1
$ octodns-sync --config idn-config.yml --doit
2024-12-02T09:33:34  [139583356670080] INFO  Manager __init__: config_file=idn-config.yml, (octoDNS 1.10.0)
2024-12-02T09:33:34  [139583356670080] INFO  Manager _config_executor: max_workers=1
2024-12-02T09:33:34  [139583356670080] INFO  Manager _config_include_meta: include_meta=False
2024-12-02T09:33:34  [139583356670080] INFO  Manager _config_enable_checksum: enable_checksum=False
2024-12-02T09:33:34  [139583356670080] INFO  Manager _config_auto_arpa: auto_arpa=False
2024-12-02T09:33:34  [139583356670080] INFO  Manager __init__: global_processors=[]
2024-12-02T09:33:34  [139583356670080] INFO  Manager __init__: global_post_processors=[]
2024-12-02T09:33:35  [139583356670080] INFO  Manager __init__: provider=zonefile-in (octodns_bind 0.0.6)
2024-12-02T09:33:35  [139583356670080] INFO  Manager __init__: provider=zonefile-out (octodns_bind 0.0.6)
2024-12-02T09:33:35  [139583356670080] INFO  Manager sync: eligible_zones=[], eligible_targets=[], dry_run=False, force=False, plan_output_fh=<stdout>, checksum=None
2024-12-02T09:33:35  [139583356670080] INFO  Manager sync:     sources=['zonefile-in']
2024-12-02T09:33:35  [139583356670080] INFO  Manager sync:   dynamic zone=*, sources=None
2024-12-02T09:33:35  [139583356670080] INFO  Manager sync:     adding dynamic zone=xn--bcher-kva.example.
2024-12-02T09:33:35  [139583356670080] INFO  Manager sync:   zone=bücher.example.
2024-12-02T09:33:35  [139583356670080] INFO  Manager sync:     sources=['zonefile-in']
2024-12-02T09:33:35  [139583356670080] INFO  Manager sync:     processors=[]
2024-12-02T09:33:35  [139583356670080] INFO  Manager sync:     targets=['zonefile-out']
2024-12-02T09:33:35  [139583356670080] INFO  ZoneFileProvider[zonefile-in] populate:   found 1 records
2024-12-02T09:33:35  [139583356670080] INFO  ZoneFileProvider[zonefile-out] plan: desired=bücher.example.
2024-12-02T09:33:35  [139583356670080] INFO  ZoneFileProvider[zonefile-out] populate:   found 0 records
2024-12-02T09:33:35  [139583356670080] WARNING ZoneFileProvider[zonefile-out] root NS record supported, but no record is configured for bücher.example.
2024-12-02T09:33:35  [139583356670080] INFO  ZoneFileProvider[zonefile-out] plan:   Creates=1, Updates=0, Deletes=0, Existing Records=0
2024-12-02T09:33:35  [139583356670080] INFO  Plan 
********************************************************************************
* bücher.example.
********************************************************************************
* zonefile-out (ZoneFileProvider)
*   Create Zone<bücher.example.>
*   Create <ARecord A 3600, bücher.example., ['127.0.0.1']> ()
*   Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
********************************************************************************


2024-12-02T09:33:35  [139583356670080] INFO  ZoneFileProvider[zonefile-out] apply: making 1 changes to bücher.example.
2024-12-02T09:33:35  [139583356670080] WARNING ZoneFileProvider[zonefile-out] _primary_nameserver: unable to find a primary_nameserver for bücher.example., using placeholder
2024-12-02T09:33:35  [139583356670080] INFO  Manager sync:   1 total changes
$ ls -l output/
total 4
-rw-rw-r-- 1 as as 228 Dec  2 09:33 bücher.example.
$ cat output/bücher.example. 
$ORIGIN bücher.example.

@ 3600 IN SOA ns.bücher.example. webmaster.bücher.example. (
    1733124815 ; Serial
    3600 ; Refresh
    600 ; Retry
    604800 ; Expire
    3600 ; NXDOMAIN ttl
)

@     3600 IN A        127.0.0.1

```

Discussion:
@ross: > Is this intentional?

Probably not, likely something that wasn't specifically targeted and just did the default. I take it that bind file format wants the idna form?

It should just be a matter of switching to use the encoded name here:

https://github.com/octodns/octodns-bind/blob/b974df8f06ce5dfd205ec5042ccd97aa75a74929/octodns_bind/__init__.py#L301

Though it probably also makes sense to include a comment with the decoded version for human consumption.

I'm also wondering what would happen with the `AxfrPopulate` setups. @github-actions: This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.@asalmela: I've used the following patch to handle this issue locally, but my python skills were not quite enough to handle the tests and getting a proper pull request out of this before I had other things to do.

```diff
diff --git a/octodns_bind/__init__.py b/octodns_bind/__init__.py
index 1ab8837..af54cd5 100644
--- a/octodns_bind/__init__.py
+++ b/octodns_bind/__init__.py
@@ -18,6 +18,7 @@ from dns import tsigkeyring
 from dns.exception import DNSException
 from dns.update import Update as DnsUpdate
 
+from octodns.idna import idna_encode, idna_decode
 from octodns.provider.base import BaseProvider
 from octodns.record import Create, Record, Rr, Update
 from octodns.source.base import BaseSource
@@ -268,7 +269,7 @@ class ZoneFileProvider(RfcPopulate, BaseProvider):
 
     def _apply(self, plan):
         desired = plan.desired
-        name = desired.decoded_name
+        name = desired.name
 
         if not isdir(self.directory):
             makedirs(self.directory)
@@ -279,7 +280,7 @@ class ZoneFileProvider(RfcPopulate, BaseProvider):
         filename = join(self.directory, f'{name[:-1]}{self.file_extension}')
         with open(filename, 'w') as fh:
             template = Template(
-                '''$$ORIGIN $zone_name
+                '''$$ORIGIN $zone_name$zone_name_comment
 
 @ $default_ttl IN SOA $primary_nameserver $hostmaster_email (
     $serial ; Serial
@@ -293,12 +294,17 @@ class ZoneFileProvider(RfcPopulate, BaseProvider):
             )
 
             primary_nameserver = self._primary_nameserver(name, records)
+            if idna_encode(name) != idna_decode(name):
+                zone_name_comment = f' ; {idna_decode(name)}'
+            else:
+                zone_name_comment = ''
             fh.write(
                 template.substitute(
                     {
                         'hostmaster_email': self._hostmaster_email(name),
                         'serial': self._serial(),
-                        'zone_name': name,
+                        'zone_name': idna_encode(name),
+                        'zone_name_comment': zone_name_comment,
                         'default_ttl': self.default_ttl,
                         'primary_nameserver': primary_nameserver,
                         'refresh': self.refresh,
@@ -326,8 +332,13 @@ class ZoneFileProvider(RfcPopulate, BaseProvider):
                         name = ''
                     else:
                         prev_name = name
+                    if idna_encode(name) != idna_decode(name):
+                        name = idna_encode(name)
+                        comment = f' ; {idna_decode(name)}'
+                    else:
+                        comment = ''
                     fh.write(
-                        f'{name:<{longest_name}} {record.ttl:8d} IN {record._type:<8} {value}\n'
+                        f'{name:<{longest_name}} {record.ttl:8d} IN {record._type:<8} {value}{comment}\n'
                     )
 
         self.log.debug(

```

Solution patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 29ac358..9ad85ca 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,6 +1,8 @@
 ## v0.0.8 - 20??-??-?? - ???
 
 - Split long TXT values using chunked_value before writing
+- Correctly handle utf-8 names and zones by writing out idna encoded values and
+  using idna encoded filenames
 - When writing RFC2317 zones, convert "/" to "-" in the filename.
 
 ## v0.0.7 - 2025-01-17 - Back to the base
diff --git a/docker-compose.yml b/docker-compose.yml
index 781ed29..b5febe6 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1,5 +1,3 @@
-version: "2.4"  # optional since v1.27.0
-
 services:
   bind9:
     # https://hub.docker.com/r/internetsystemsconsortium/bind9
@@ -17,6 +15,9 @@ services:
       - type: bind
         source: ./docker/var/lib/bind/db.exxampled.com
         target: /var/lib/bind/db.exxampled.com
+      - type: bind
+        source: ./docker/var/lib/bind/db.xn--fa-hia.de.
+        target: /var/lib/bind/db.xn--fa-hia.de.
 networks:
   dns:
     enable_ipv6: true
diff --git a/docker/etc/bind/named.conf b/docker/etc/bind/named.conf
index 4b22bf5..97f4010 100644
--- a/docker/etc/bind/named.conf
+++ b/docker/etc/bind/named.conf
@@ -27,6 +27,14 @@ zone "exxampled.com." {
   allow-update { key octodns.exxampled.com.; };
 };
 
+zone "xn--fa-hia.de." {
+  type master;
+  file "/var/lib/bind/db.xn--fa-hia.de.";
+  notify explicit;
+  allow-transfer { key octodns.exxampled.com.; };
+  allow-update { key octodns.exxampled.com.; };
+};
+
 # logging
 logging {
   channel stdout {
diff --git a/docker/var/lib/bind/db.xn--fa-hia.de. b/docker/var/lib/bind/db.xn--fa-hia.de.
new file mode 100644
index 0000000..2e620d8
--- /dev/null
+++ b/docker/var/lib/bind/db.xn--fa-hia.de.
@@ -0,0 +1,18 @@
+$ORIGIN xn--fa-hia.de.
+
+; Zone name: faß.de. / xn--fa-hia.de.
+@ 3600 IN SOA ns1.xn--fa-hia.de. webmaster.xn--fa-hia.de. (
+    1742254963 ; Serial
+    3600 ; Refresh
+    600 ; Retry
+    604800 ; Expire
+    3600 ; NXDOMAIN ttl
+)
+
+; Name: faß.de.
+@                  86400 IN MX       10 mx.xn--fa-hia.de.
+                   86400 IN NS       ns1.xn--fa-hia.de.
+mx                 86400 IN A        127.0.0.1
+ns1                86400 IN A        127.0.0.1
+; Name: déjà-vu.faß.de.
+xn--dj-vu-sqa5d    86400 IN A        127.0.0.1
diff --git a/octodns_bind/__init__.py b/octodns_bind/__init__.py
index b2de62c..95be6f4 100644
--- a/octodns_bind/__init__.py
+++ b/octodns_bind/__init__.py
@@ -270,7 +270,6 @@ def _serial(self):
 
     def _apply(self, plan):
         desired = plan.desired
-        name = desired.decoded_name
 
         if not isdir(self.directory):
             makedirs(self.directory)
@@ -278,15 +277,18 @@ def _apply(self, plan):
         records = sorted(c.record for c in plan.changes)
         longest_name = self._longest_name(records)
 
+        name = desired.name
         filename = join(
             self.directory,
             f'{name[:-1].replace("/", "-")}{self.file_extension}',
         )
         with open(filename, 'w') as fh:
+            fh.write(f'$ORIGIN {name}\n\n')
+            utf8_name = desired.decoded_name
+            if name != utf8_name:
+                fh.write(f'; Zone name: {utf8_name}\n')
             template = Template(
-                '''$$ORIGIN $zone_name
-
-@ $default_ttl IN SOA $primary_nameserver $hostmaster_email (
+                '''@ $default_ttl IN SOA $primary_nameserver $hostmaster_email (
     $serial ; Serial
     $refresh ; Refresh
     $retry ; Retry
@@ -321,15 +323,18 @@ def _apply(self, plan):
                 except AttributeError:
                     values = [record.value]
                 for value in values:
-                    value = value.rdata_text
-                    if record._type in ('SPF', 'TXT'):
-                        # TXT values need to be quoted and split if longer than 255 characters
-                        value = record.chunked_value(value)
                     name = '@' if record.name == '' else record.name
                     if name == prev_name:
                         name = ''
                     else:
                         prev_name = name
+                        if name != record.decoded_name:
+                            # idna encoded, add a comment with the utf8 version
+                            fh.write(f'; Name: {record.decoded_fqdn}\n')
+                    value = value.rdata_text
+                    if record._type in ('SPF', 'TXT'):
+                        # TXT values need to be quoted and split if longer than 255 characters
+                        value = record.chunked_value(value)
                     fh.write(
                         f'{name:<{longest_name}} {record.ttl:8d} IN {record._type:<8} {value}\n'
                     )


Test patch:
diff --git a/tests/test_provider_octodns_bind.py b/tests/test_provider_octodns_bind.py
index 1967018..0376350 100644
--- a/tests/test_provider_octodns_bind.py
+++ b/tests/test_provider_octodns_bind.py
@@ -293,6 +293,7 @@ def test_apply(self, serial_mock):
     3600 ; NXDOMAIN ttl
 )
 
+; Name: unit.tests.
 @           44 IN A        1.2.3.4
             43 IN NS       ns1.unit.tests.
             43 IN NS       ns2.unit.tests.
@@ -330,9 +331,66 @@ def test_apply(self, serial_mock):
     3601 ; NXDOMAIN ttl
 )
 
+; Name: unit.tests.
 @         43 IN NS       ns1.unit.tests.
           43 IN NS       ns2.unit.tests.
 txt       45 IN TXT      "hello \\" world"
+''',
+                    fh.read(),
+                )
+
+    @patch('octodns_bind.ZoneFileProvider._serial')
+    def test_utf8(self, serial_mock):
+        serial_mock.side_effect = [424344]
+
+        with TemporaryDirectory() as td:
+            provider = ZoneFileProvider('target', td.dirname)
+
+            desired = Zone('faß.de.', [])
+
+            ns = Record.new(
+                desired,
+                '',
+                {
+                    'type': 'NS',
+                    'ttl': 43,
+                    'values': ('ns1.xn--fa-hia.de.', 'ns2.xn--fa-hia.de.'),
+                },
+            )
+            desired.add_record(ns)
+
+            # utf-8 zone name
+            txt = Record.new(
+                desired,
+                # utf-8 name
+                'déjà-vu',
+                {'type': 'TXT', 'ttl': 45, 'value': 'hello world'},
+            )
+            desired.add_record(txt)
+
+            changes = [Create(txt), Create(ns)]
+            plan = Plan(None, desired, changes, True)
+            provider._apply(plan)
+            # file written out with idna name
+            with open(join(td.dirname, 'xn--fa-hia.de.')) as fh:
+                self.maxDiff = 99999
+                self.assertEqual(
+                    '''$ORIGIN xn--fa-hia.de.
+
+; Zone name: faß.de.
+@ 3600 IN SOA ns1.xn--fa-hia.de. webmaster.xn--fa-hia.de. (
+    424344 ; Serial
+    3600 ; Refresh
+    600 ; Retry
+    604800 ; Expire
+    3600 ; NXDOMAIN ttl
+)
+
+; Name: faß.de.
+@                     43 IN NS       ns1.xn--fa-hia.de.
+                      43 IN NS       ns2.xn--fa-hia.de.
+; Name: déjà-vu.faß.de.
+xn--dj-vu-sqa5d       45 IN TXT      "hello world"
 ''',
                     fh.read(),
                 )
@@ -384,6 +442,7 @@ def test_apply_rfc2317(self, serial_mock):
     3600 ; NXDOMAIN ttl
 )
 
+; Name: 0/25.10.10.10.in-addr.arpa.
 @        42 IN NS       ns.unit.tests.
 10       42 IN PTR      target.unit.tests.
 """,