Change:
Describe the bug When upgrading from 20.03 to 20.09, acme started failing for 5 of my domains (but not all of them). To Reproduce start acme- Expected behavior No failure Additional context Add any other context about the problem here. Notify maintainers cc Metadata system: host os: multi-user?: sandbox: version: channels(root): channels(symphorien): nixpkgs: Maintainer information:
I hit the same bug as well and fixed it on my machines with the following workaround:
Thanks for reposting that . Think I will extend with some info on how to deal with this error.
I wiped and now all certificate services fail:
Right - the tests for whether to run or don't check the accounts directory, I will fix that now. That said, I don't know if ACME fairs well if you delete your account details and keep your certificates. I'm pretty sure you need to use the same account to issue a renew request for existing certificates.
Do you mean that I should nuke all of /var/lib/acme ?
Pretty much, yeah. Personally I would move it somewhere else instead of flat-out delete ;) Then run .
It works.
Awesome. I have opened a PR there which will resolve this issue. In particular, I made it check if the accounts directory is empty, which hopefully negates the need to wipe out your entire /var/lib/acme directory. I also some documentation on the process.
I am also running into this problem. The workaround works, but the account key was generated just yesterday and nothing looks corrupted (JSON and EC key). Today, I was trying to enroll a new domain using the same account. There may be something more fishy around this. Unsure how to debug this.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
prevents renewal services to be run at the same time. I'm unsure as to how to test if it fixes the issue.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
FWIW I don't think that renewing more than one cert at a time is a factor in this bug. I only had a single cert fail when upgrading to NixOS 20.09 and running the single systemd service by itself still failed. It appears this will be fixed by That said, linking all of the renewal services together would be nice so that if one fails the others won't be tried. If you have 5 or more failures you basically lock yourself out of LE for an hour.
This PR does not do that. If a certificate fails, the next ones will be tried as well.
and others, I have done some work in to prevent multiple simultaneous account creations, and reduced the usage of tmpfiles so that file creation is more dependable/reliable. If you could give it a shot and report back I'd be interested to hear if it works :) You may need to run first to get rid of any potential broken stuff. Bare in mind this will delete your existing certs, keys and account data for that domain.
The issue still exists on 20.09 (I could not find any backport), and it happened to me this very morning.
Also happened to me just now on 20.09. The workaround from did not fix it for me.
I did that and it worked, but this may hit the ratelimit if you have too many certificates.
Just so it's well tracked, the solution merged to master was backported to 20.09 in
On another server than the one from , I still get: In this case, however, the solution from in did the trick: This made it work. In case it helps investigate, as shown I had 2 accounts in there: and . However, they had the same config, except from different IDs: After the fix, comparing the new account (same ID as for one of the s was generated, and the first 3 lines of the private key files are different): Why isn't there anything printed in the error after ? Shouldn't there be a URL there, or is it the empty string?
This makes sense if you have just applied the backported changes, since it changed the way the accountID hash is generated. As for why you got JWS verification error, it still doesn't make sense to me. Have you got multiple certs configured with the same account on either host? At this point, I feel the best investment of my time is to go to upstream on this issue and ask them about it, but it's difficult when I cannot reproduce this in any of my own setups.
No, on this server I'm working with only 1 domain and no subdomain. I then use the cert for this domain for multiple NixOS services (e.g. , , and now looking into configuring ). That sounds like a good idea. If it would be helpful, we can inspect the state of my server(s) together in a screenshare. Ping me as on on IRC (e.g. I'm available right now). Note that I cannot properly "reproduce" though (applying fix makes the error go away permanently), but if there's more state we could inspect, I can offer that.
and I debugged in a screenshare; writeup so far: We searched for the error message in . From a quick grep, it could only be this one: Further down we found: This looks wrong; why would you append a string if you know it has length 0? Probably a typo for . So we planned to reproduce it with a patched version of lego. We moved the problematic account back into place: did not renew because the cert was too new. To force certifical renewal, we did: and again. That reproduced the error successfully: Fixing the bug above just showed that was indeed the empty string, so no further info from that. The Response body was: So while the fixed bug is nice (upstream PR ), it doesn't tell us anything new, we have to debug further why LetsEncrypt gives a verification error.
Ok! I figured out how to reproduce this from a working setup to the LE staging server a new certificate to Copy to from the prefixed folder either or the private key (one of them, either works) to the unprefixed TL;DR If you end up in a situation where you have the wrong key for a given account, you get the exact error being shared around here. Why this is happening is what really confuses me. This has been even when you only have 1 certificate involved, which rules out a multi-cert race condition. By virtue of the hashed folders, you can never really end up with stale data in your accounts folder. Anyone that has an accounts directory that has this issue - can you please cat the key in your accounts folder and check it is not malformed? Also, can you compare the modify times of the and the key file (they should be fairly close)? That's my best guess right now as to what the cause is. What's nicer is that I have_ found an elegant solution: If your key is not corrupt, then you can re-request the belonging account ID. To quote the : To do this, simply delete the in your accounts folder, and run a renew.
Deleting the file did not fix the issue for me. Here is the account directory before deleting the file: Before deleting the file, running the service produced: After deleting the file and running the service again:
Also did not work for me.
Just straight up deleting helped though. Think about what you're doing before you do this, but for me it didn't matter if things didn't work. Fortunately though, it did.
That actually did work for me, though hopefully I was in the correct situation: After this, was recreated with a different account ID, and the key remained the same. For what it's worth, the numeric ID is 3 lower in value than the backup. And as you can see, there was a little over 4 hours inbetween. For some background, this is on EC2 and we provisioned this server today. We activated a configuration that requested 10 certificates in one go, and all went well there. Later, we had to change one of the nginx virtual hosts, so caused a new certificate request there. That one failed with the JWS verification error. So ACME configuration was only changed twice: in the initial activation, and the later change. (Other activations did cause the services to rerun, just do nothing, because certs were up-to-date.) Unfortunately, this is not a 1-certificate-scenario, and so doesn't rule out parallelism. This machine is currently on nixpkgs on branch nixos-20.09. I would expect an actually corrupt key on disk would cause a different error, because Lego would not be able to sign the request? It's my guess that we can rule out a corrupt key file.
Thanks for the background info. Fortunately I have definitely fixed any parallelism issues in :) If that was the cause in your case, it should be resolved soon. The fact that your ID was 3 lower confirms it IMO. I would suspect the same too, but I haven't looked into how Lego uses that key exactly. I would hope it gives a different error. I came up with another possible scenario where this solution may not work (at least, deleting the would not work). AFAIK the account keys are generated by lego. If the key is generated, written to disk, then account creation fails, then you end up with a key with no account associated with it. This could be the issue in your case. Try removing the key instead of the and report back what happens :)
Actually come to think of it there's no way for lego to get the key. I suspect it will still work, but your account ID will change. That's what happened when I tried it.
If that last comment was directed to me... If my account ID is going to change should I just archive and start from scratch?
I guess so, emphasis on the archive since right now I don't know what else to investigate as a possible cause.
My server with multiple certs, the one where all the acme services seemed to resolve themselves, started failing again. Nothing on this server changed. So I decided to archive the directory and start from zero. When running one of the renewal services I get the following error: I've started receiving Let's Encrypt emails stating that some of my certs are about to expire. At this point all of my servers are failing to renew all certifications because of acme service errors. At this point I don't know what to do and I'm only a few days away from a production outage due to expired certs. Please advise.
Can you send the logs from the last good renewal of one of your certs? Also can you indicate: Whether you use one email/account for all certs or multiple accounts Whether /var/lib/acme and /tmp are on different file systems Also you need to run if you blow away your whole /var/lib/acme
The services have been failing long enough where I no longer have logs that go back that far. As far as accounts go, I use one email address across all of my certs. Using was the missing piece that I needed. I was able to recreate the certs on one of my servers successfully. I'll start working on the other servers next. Thank you for your quick response.
Glad you got sorted. I am constantly surprised by people's encounters with this bug. I personally have a lot of hosts using the module; one with over 20 certs all using one account, and I have never hit some of these common issues - which I hope is a bit of good news. Once you rebuild your /var/lib/acme it tends to go away for good but I've not found a good reason why it breaks in the first place. Hopefully the work in the other PR will help people.
Is there any solution or workaround for this "JWS verification error"? Even removing + running does not help in any way. There's always the error: Any help would be appreciated.
FYI I had to patch in my local nixpkgs, backup/remove , run and run the service to finally get it to work.
Will there be any fix for this problem in NixOS 20.09.? Thanks. Can you tell me how to patch this?
This doesnt make sense. I removed all tmpfiles usage in so that couldn't have an impact on any resolution. However, if that patch does solve the issue then I will look into getting a backport merged
Sorry if this is just a coincidence then. FYI I was running at + the PR mentioned above when I finally got rid of the ACME errors.
I have opened a backport for in :)
This is fixed now!
To whom it may concern: I still had to remove/backup to get the update to work.
There's not much that can be done about this, but I am hopeful that the latest set of PRs will stop the issue occurring again. Out of interest though, did you get this error specifically (JWS verification) or something else?
Just had this issue today on unstable. To get it working again I did the following Lots of good hints in this thread here. Thanks!
To anyone else that comes across this thread: I also wrote which detail only deleting the accounts folder to resolve this issue. From previous testing in this thread, this was usually all that was required. Failing that, could you please try these commands before the above options in the thread as I am curious if it will work: This will delete the certs + accounts data for the failing renewal. It's a little less destructive if you're only having a problem with one cert than recreating the whole of /var/lib/acme

Discussion:
mkHash = with builtins; val: substring 0 20 (hashString "sha256" val); certDir = mkHash hashData; domainHash = mkHash "${concatStringsSep " " extraDomains} ${data.domain}"; <del> othersHash = mkHash "${toString acmeServer} ${data.keyType}"; </del> <ins> othersHash = mkHash "${toString acmeServer} ${data.keyType} ${data.email}"; </ins> accountDir = "/var/lib/acme/.lego/accounts/" + othersHash; protocolOpts = if useDns then (

echo '${domainHash}' > domainhash.txt # Check if we can renew <del> if [ -e 'certificates/${keyName}.key' -a -e 'certificates/${keyName}.crt' ]; then </del> <ins> # Certificates and account credentials must exist if [ -e 'certificates/${keyName}.key' -a -e 'certificates/${keyName}.crt' -a "$(ls -1 accounts)" ]; then </ins> # When domains are updated, there's no need to do a full # Lego run, but it's likely renew won't work if days is too low.

ones. </para> </section> <ins> <section xml:id="module-security-acme-fix-jws"> <title>Fixing JWS Verification error</title> <para> It is possible that your account credentials file may become corrupt and need to be regnerated. In this scenario lego will produce the error <literal>JWS verification error</literal>. The solution is to simply delete the associated accounts file and re-run the affected service(s). </para> <programlisting> # Find the accounts folder for the certificate systemctl cat acme-example.com.service | grep -Po 'accounts/[^:]*' export accountdir="$(!!)" # Move this folder to some place else mv /var/lib/acme/.lego/$accountdir{,.bak} # Recreate the folder using systemd-tmpfiles systemd-tmpfiles --create # Get a new account and reissue certificates # Note: Do this for all certs that share the same account email address systemctl start acme-example.com.service </programlisting> </section> </ins> </chapter>