Change:
Describe the bug An recent change on release-20.09 broke acme on my system: It's trying to chown a path that's currently owned by root: I track nixpkgs versions in my setup, so the working state is and the broken state is . To Reproduce (Not sure if this is easily reproducible, but...) Have a config like and do an upgrade from NixOS 20.09 @ git commit to . Expected behavior That services that used to work don't get permission error and fail on the stable branch. Additional context I rolled back to the known good version, that failed now(!), I rolled forward again and manually fixed up the permissions. Now I'm getting this error: And later: I manually restarted the acme service. Now everything seems fine. Notify maintainers <!-- Please @ people who are in the list of the offending package or module. If in doubt, check for whoever last touched something. --CC CC due to their acme commits on release-20.09. Metadata Maintainer information:
seems to have removed a log message that I would have found helpful here: That changes the error from a "this module is broken" to "this module is guiding me to do a manual data migration". That's a big deal. Luckily did add , which was very helpful.
I'm glad you found the old error message - that is indeed the issue here. .well-known + .well-known/acme-challenge need to be owned by acme + the configured group. I will add back in the echo in my next PR. Thanks for pointing that out.
In a way it's weird that the code tries to do as non-root user, because it'll always fail if the ownership is not already correct. But maybe the point is just to verify the ownership? Thanks, adding back that message will help. Maybe even print the command the user should run ( )?
Verifying the owner is correct, but setting the group is the main intention since it can be changed between rebuilds (and will work in normal circumstances). I guess would be clearer here now that I think about it, and would still raise and appropriate error if the owner is not acme. I'll add it to the list Thanks!
From an end-user perspective, I wonder why the data migration isn't happening automatically. Why cannot NixOS run the needed command as root? Is there risk of data loss?
I stumbled upon this today. I'm slightly confused: Why does the module need ownership of ? Shouldn't ownership of be enough? I had the ownership of the acme-challenge directory correct, but .well-known was owned by root, so it broke for me. But my understanding of the challenge is that it should only write/delete files in the acme-challenge subdirectory.
The reason .well-known is also chown'ed is that lego (the ACME client that the module uses) tries to create acme-challenge every time, and will throw a permissions error before checking if it exists. I thought I had a copy of the error in comments somewhere but I can't find it now, so I will test it once again to make sure. As for a more permanent fix, I think I will move these ownership fixes into the acme-fixperms.service which runs as root.
After some testing and some , I found that I don't actually need to fix .well-known any more. I'm almost certain this wasn't the case at one point, but alas removing it will resolve issues here. I have stripped the problematic part of the renewal script down to its bare minimum - it now only runs the following commands: The chgrp is required as the configured group can be changed between runs. Arguably this could be removed too if you assume that acme will always own this folder, and the changes I've made . (Set UMask to 0022 for the service) make it world readable. In an abundance of caution, I am going to keep it around for now.
I completely wiped my directory but starting acme fails with: This is on 20.09. does not exist yet when this is run. Is this the same issue in a dfiferent flavor? i don't have any acme related state on the machine and this is the entire config:
Let's leave this open until the fix is backported to release-20.09.
Sorry it took me so long... busy week. PR is open. If someone like could give it a test and see if it fixes their issues that would be great.
Backports merged, closing issue.

Discussion:
Type = "oneshot"; User = "acme"; Group = mkDefault "acme"; <del> UMask = 0023; </del> <ins> UMask = 0022; </ins> StateDirectoryMode = 750; ProtectSystem = "full"; PrivateTmp = true;

set -euxo pipefail ${optionalString (data.webroot != null) '' <del> # Ensure the webroot exists mkdir -p '${data.webroot}/.well-known/acme-challenge' chown 'acme:${data.group}' ${data.webroot}/{.well-known,.well-known/acme-challenge} </del> <ins> # Ensure the webroot exists. Fixing group is required in case configuration was changed between runs. # Lego will fail if the webroot does not exist at all. ( mkdir -p '${data.webroot}/.well-known/acme-challenge'  && chgrp '${data.group}' ${data.webroot}/.well-known/acme-challenge ) || ( echo 'Please ensure ${data.webroot}/.well-known/acme-challenge exists and is writable by acme:${data.group}'  && exit 1 ) </ins> ''} echo '${domainHash}' > domainhash.txt

def check_connection(node, domain, retries=3): <del> assert retries >= 0 </del> <ins> assert retries >= 0, f"Failed to connect to https://{domain}" </ins> result = node.succeed( "openssl s_client -brief -verify 2 -CAfile /tmp/ca.crt"

for line in result.lower().split("n"): if "verification" in line and "error" in line: <del> time.sleep(1) </del> <ins> time.sleep(3) </ins> return check_connection(node, domain, retries - 1) def check_connection_key_bits(node, domain, bits, retries=3): <del> assert retries >= 0 </del> <ins> assert retries >= 0, f"Did not find expected number of bits ({bits}) in key" </ins> result = node.succeed( "openssl s_client -CAfile /tmp/ca.crt"

print("Key type:", result) if bits not in result: <del> time.sleep(1) </del> <ins> time.sleep(3) </ins> return check_connection_key_bits(node, domain, bits, retries - 1) def check_stapling(node, domain, retries=3): <del> assert retries >= 0 </del> <ins> assert retries >= 0, "OCSP Stapling check failed" </ins> # Pebble doesn't provide a full OCSP responder, so just check the URL result = node.succeed(

print("OCSP Responder URL:", result) if "${caDomain}:4002" not in result.lower(): <del> time.sleep(1) </del> <ins> time.sleep(3) </ins> return check_stapling(node, domain, retries - 1) <ins> def download_ca_certs(node, retries=5): assert retries >= 0, "Failed to connect to pebble to download root CA certs" exit_code, _ = node.execute("curl https://${caDomain}:15000/roots/0 > /tmp/ca.crt") exit_code_2, _ = node.execute( "curl https://${caDomain}:15000/intermediate-keys/0 >> /tmp/ca.crt" ) if exit_code + exit_code_2 > 0: time.sleep(3) return download_ca_certs(node, retries - 1) </ins> client.start() dnsserver.start()

acme.wait_for_unit("network-online.target") acme.wait_for_unit("pebble.service") <del> client.succeed("curl https://${caDomain}:15000/roots/0 > /tmp/ca.crt") client.succeed("curl https://${caDomain}:15000/intermediate-keys/0 >> /tmp/ca.crt") </del> <ins> download_ca_certs(client) </ins> with subtest("Can request certificate with HTTPS-01 challenge"): webserver.wait_for_unit("acme-finished-a.example.test.target")

check_issuer(webserver, "a.example.test", "pebble") check_connection(client, "a.example.test") <ins> with subtest("Certificates and accounts have safe + valid permissions"): group = "${nodes.webserver.config.security.acme.certs."a.example.test".group}" webserver.succeed( f"test $(stat -L -c "%a %U %G" /var/lib/acme/a.example.test/* | tee /dev/stderr | grep '640 acme {group}' | wc -l) -eq 5" ) webserver.succeed( f"test $(stat -L -c "%a %U %G" /var/lib/acme/.lego/a.example.test/**/* | tee /dev/stderr | grep '640 acme {group}' | wc -l) -eq 5" ) webserver.succeed( f"test $(stat -L -c "%a %U %G" /var/lib/acme/a.example.test | tee /dev/stderr | grep '750 acme {group}' | wc -l) -eq 1" ) webserver.succeed( f"test $(find /var/lib/acme/accounts -type f -exec stat -L -c "%a %U %G" {{}} ; | tee /dev/stderr | grep -v '600 acme {group}' | wc -l) -eq 0" ) </ins> with subtest("Can generate valid selfsigned certs"): webserver.succeed("systemctl clean acme-a.example.test.service --what=state") webserver.succeed("systemctl start acme-selfsigned-a.example.test.service")

assert keyhash_old == keyhash_new with subtest("Can request certificates for vhost + aliases (apache-httpd)"): <del> switch_to(webserver, "httpd-aliases") webserver.wait_for_unit("acme-finished-c.example.test.target") </del> <ins> try: switch_to(webserver, "httpd-aliases") webserver.wait_for_unit("acme-finished-c.example.test.target") except Exception as err: _, output = webserver.execute( "cat /var/log/httpd/*.log && ls -al /var/lib/acme/acme-challenge" ) print(output) raise err </ins> check_issuer(webserver, "c.example.test", "pebble") check_connection(client, "c.example.test") check_connection(client, "d.example.test")