Repository: certbot/certbot

Issue:
certbot/dns-google no longer working: OpenSSL.crypto.Error: [('asn1 encoding routines', 'ASN1_get_object', 'header too long')]
The bug report below is for v.1.9.0 in the official Docker container, though I've tried 1.8.0, as well as a manual container build (installing the python module manually) - it all comes to the same issue.

## My operating system is (include version):

Docker container: certbot/dns-google:v1.9.0

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):

Docker container: certbot/dns-google:v1.9.0

## I ran this command and it produced this output:

```
certbot certonly -n --agree-tos -m root@<redacted> -d <redacted> --dns-google --dns-google-credentials /root/certbot.json
```

Output:

```
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-google, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for <redacted>
Unsafe permissions on credentials configuration file: /root/certbot.json
Cleaning up challenges
Encountered exception during recovery: OpenSSL.crypto.Error: [('asn1 encoding routines', 'ASN1_get_object', 'header too long')]
An unexpected error occurred:
OpenSSL.crypto.Error: [('asn1 encoding routines', 'ASN1_get_object', 'header too long')]
Please see the logfiles in /var/log/letsencrypt for more details.

IMPORTANT NOTES:
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
```

No certificates are generated.

## Certbot's behavior differed from what I expected because:


## Here is a Certbot log showing the issue (if available):
###### Logs are stored in `/var/log/letsencrypt` by default. Feel free to redact domains, e-mail and IP addresses as you see fit.

```
2020-11-28 23:28:01,987:DEBUG:certbot._internal.main:certbot version: 1.9.0
2020-11-28 23:28:01,988:DEBUG:certbot._internal.main:Arguments: ['-n', '--agree-tos', '-m', '<redacted>', '-d', '<redacted>', '--dns-google', '--dns-google-credentials', '/root/certbot.json']
2020-11-28 23:28:01,988:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-google,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-11-28 23:28:02,021:DEBUG:certbot._internal.log:Root logging level set at 20
2020-11-28 23:28:02,021:INFO:certbot._internal.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-11-28 23:28:02,025:DEBUG:certbot._internal.plugins.selection:Requested authenticator dns-google and installer None
2020-11-28 23:28:02,036:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * dns-google
Description: Obtain certificates using a DNS TXT record (if you are using Google Cloud DNS for DNS).
Interfaces: IAuthenticator, IPlugin
Entry point: dns-google = certbot_dns_google._internal.dns_google:Authenticator
Initialized: <certbot_dns_google._internal.dns_google.Authenticator object at 0x7ff678e68f10>
Prep: True
2020-11-28 23:28:02,036:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_dns_google._internal.dns_google.Authenticator object at 0x7ff678e68f10> and installer None
2020-11-28 23:28:02,037:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator dns-google, Installer None
2020-11-28 23:28:02,169:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2020-11-28 23:28:02,171:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2020-11-28 23:28:02,910:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2020-11-28 23:28:02,912:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sat, 28 Nov 2020 23:28:02 GMT
Content-Type: application/json
Content-Length: 658
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "P5L3y7xb7Bo": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2020-11-28 23:28:02,914:DEBUG:acme.client:Requesting fresh nonce
2020-11-28 23:28:02,914:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
2020-11-28 23:28:03,114:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
2020-11-28 23:28:03,115:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sat, 28 Nov 2020 23:28:02 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0104Sx186beDFfbgCMUygioRmrdwmk_YXVoFxBaDuT8Hr4I
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


2020-11-28 23:28:03,115:DEBUG:acme.client:Storing nonce: 0104Sx186beDFfbgCMUygioRmrdwmk_YXVoFxBaDuT8Hr4I
2020-11-28 23:28:03,116:DEBUG:acme.client:JWS payload:
b'{\n  "contact": [\n    "mailto:<redacted>"\n  ],\n  "termsOfServiceAgreed": true\n}'
2020-11-28 23:28:03,119:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-acct:
{
  "protected": "<snip>",
  "signature": "<snip>",
  "payload": "<snip>"
}
2020-11-28 23:28:03,526:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-acct HTTP/1.1" 201 566
2020-11-28 23:28:03,527:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Date: Sat, 28 Nov 2020 23:28:03 GMT
Content-Type: application/json
Content-Length: 566
Connection: keep-alive
Boulder-Requester: 104140260
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index", <https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf>;rel="terms-of-service"
Location: https://acme-v02.api.letsencrypt.org/acme/acct/104140260
Replay-Nonce: 0103mL9kEFzBn3jSaWXz_A4YFjv1OUh8KMgFMbO1vlHkE8M
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "key": {
    "kty": "RSA",
    "n": "<snip>",
    "e": "AQAB"
  },
  "contact": [
    "mailto:<redacted>"
  ],
  "initialIp": "79.252.61.239",
  "createdAt": "2020-11-28T23:28:03.259843092Z",
  "status": "valid"
}
2020-11-28 23:28:03,527:DEBUG:acme.client:Storing nonce: 0103mL9kEFzBn3jSaWXz_A4YFjv1OUh8KMgFMbO1vlHkE8M
2020-11-28 23:28:03,528:DEBUG:certbot._internal.reporter:Reporting to user: Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal.
2020-11-28 23:28:03,529:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x7ff678e620d0>)>), contact=('mailto:<redacted>',), agreement=None, status='valid', terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/104140260', new_authzr_uri=None, terms_of_service='https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf'), dd557eb76f6d685702c405615ec71441, Meta(creation_dt=datetime.datetime(2020, 11, 28, 23, 28, 3, tzinfo=<UTC>), creation_host='422feda5dbe4', register_to_eff=None))>
2020-11-28 23:28:03,530:INFO:certbot._internal.main:Obtaining a new certificate
2020-11-28 23:28:03,567:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0000_key-certbot.pem
2020-11-28 23:28:03,572:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0000_csr-certbot.pem
2020-11-28 23:28:03,574:DEBUG:acme.client:JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "<redacted>"\n    }\n  ]\n}'
2020-11-28 23:28:03,576:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "<snip>",
  "signature": "<snip>",
  "payload": "<snip>"
}
2020-11-28 23:28:04,141:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 353
2020-11-28 23:28:04,142:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Date: Sat, 28 Nov 2020 23:28:03 GMT
Content-Type: application/json
Content-Length: 353
Connection: keep-alive
Boulder-Requester: 104140260
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-v02.api.letsencrypt.org/acme/order/104140260/6461869465
Replay-Nonce: 0104PKG9pFUD20pXqIl-7SwuL2uN8gEXOWycc6C5fVy3noI
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2020-12-05T23:28:03.849596502Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "<redacted>"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/8936504574"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/104140260/6461869465"
}
2020-11-28 23:28:04,142:DEBUG:acme.client:Storing nonce: 0104PKG9pFUD20pXqIl-7SwuL2uN8gEXOWycc6C5fVy3noI
2020-11-28 23:28:04,144:DEBUG:acme.client:JWS payload:
b''
2020-11-28 23:28:04,146:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/8936504574:
{
  "protected": "<snip>",
  "signature": "<snip>",
  "payload": ""
}
2020-11-28 23:28:04,347:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/8936504574 HTTP/1.1" 200 800
2020-11-28 23:28:04,349:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sat, 28 Nov 2020 23:28:04 GMT
Content-Type: application/json
Content-Length: 800
Connection: keep-alive
Boulder-Requester: 104140260
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0103q5phzGM5q3hPstm1nps1I3vgOancQ4vKfb9RCmbHMn4
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "<redacted>"
  },
  "status": "pending",
  "expires": "2020-12-05T23:28:03Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/8936504574/cbooPQ",
      "token": "i2IXqH3h5yapehFJ1cs9b2TZ1i8z8DiMDkWxhGlRBeY"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/8936504574/sXcZgA",
      "token": "i2IXqH3h5yapehFJ1cs9b2TZ1i8z8DiMDkWxhGlRBeY"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/8936504574/cEuBCg",
      "token": "i2IXqH3h5yapehFJ1cs9b2TZ1i8z8DiMDkWxhGlRBeY"
    }
  ]
}
2020-11-28 23:28:04,350:DEBUG:acme.client:Storing nonce: 0103q5phzGM5q3hPstm1nps1I3vgOancQ4vKfb9RCmbHMn4
2020-11-28 23:28:04,351:INFO:certbot._internal.auth_handler:Performing the following challenges:
2020-11-28 23:28:04,351:INFO:certbot._internal.auth_handler:dns-01 challenge for <redacted>
2020-11-28 23:28:04,353:WARNING:certbot.plugins.dns_common:Unsafe permissions on credentials configuration file: /root/certbot.json
2020-11-28 23:28:04,358:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 70, in handle_authorizations
    resps = self.auth.perform(achalls)
  File "/usr/lib/python3.8/site-packages/certbot/plugins/dns_common.py", line 57, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/usr/lib/python3.8/site-packages/certbot_dns_google/_internal/dns_google.py", line 70, in _perform
    self._get_google_client().add_txt_record(domain, validation_name, validation, self.ttl)
  File "/usr/lib/python3.8/site-packages/certbot_dns_google/_internal/dns_google.py", line 76, in _get_google_client
    return _GoogleClient(self.conf('credentials'))
  File "/usr/lib/python3.8/site-packages/certbot_dns_google/_internal/dns_google.py", line 88, in __init__
    credentials = ServiceAccountCredentials.from_json_keyfile_name(account_json, scopes)
  File "/usr/lib/python3.8/site-packages/oauth2client/service_account.py", line 221, in from_json_keyfile_name
    return cls._from_parsed_json_keyfile(client_credentials, scopes,
  File "/usr/lib/python3.8/site-packages/oauth2client/service_account.py", line 185, in _from_parsed_json_keyfile
    signer = crypt.Signer.from_string(private_key_pkcs8_pem)
  File "/usr/lib/python3.8/site-packages/oauth2client/_openssl_crypt.py", line 119, in from_string
    pkey = crypto.load_pkcs12(key, password).get_privatekey()
  File "/usr/lib/python3.8/site-packages/OpenSSL/crypto.py", line 3177, in load_pkcs12
    _raise_current_error()
  File "/usr/lib/python3.8/site-packages/OpenSSL/_util.py", line 57, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.crypto.Error: [('asn1 encoding routines', 'ASN1_get_object', 'header too long')]

2020-11-28 23:28:04,358:DEBUG:certbot._internal.error_handler:Calling registered functions
2020-11-28 23:28:04,358:INFO:certbot._internal.auth_handler:Cleaning up challenges
2020-11-28 23:28:04,361:ERROR:certbot._internal.error_handler:Encountered exception during recovery: OpenSSL.crypto.Error: [('asn1 encoding routines', 'ASN1_get_object', 'header too long')]
2020-11-28 23:28:04,361:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/main.py", line 1362, in main
    return config.func(config, plugins)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/main.py", line 1243, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/main.py", line 122, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/client.py", line 418, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/client.py", line 351, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/client.py", line 398, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 70, in handle_authorizations
    resps = self.auth.perform(achalls)
  File "/usr/lib/python3.8/site-packages/certbot/plugins/dns_common.py", line 57, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/usr/lib/python3.8/site-packages/certbot_dns_google/_internal/dns_google.py", line 70, in _perform
    self._get_google_client().add_txt_record(domain, validation_name, validation, self.ttl)
  File "/usr/lib/python3.8/site-packages/certbot_dns_google/_internal/dns_google.py", line 76, in _get_google_client
    return _GoogleClient(self.conf('credentials'))
  File "/usr/lib/python3.8/site-packages/certbot_dns_google/_internal/dns_google.py", line 88, in __init__
    credentials = ServiceAccountCredentials.from_json_keyfile_name(account_json, scopes)
  File "/usr/lib/python3.8/site-packages/oauth2client/service_account.py", line 221, in from_json_keyfile_name
    return cls._from_parsed_json_keyfile(client_credentials, scopes,
  File "/usr/lib/python3.8/site-packages/oauth2client/service_account.py", line 185, in _from_parsed_json_keyfile
    signer = crypt.Signer.from_string(private_key_pkcs8_pem)
  File "/usr/lib/python3.8/site-packages/oauth2client/_openssl_crypt.py", line 119, in from_string
    pkey = crypto.load_pkcs12(key, password).get_privatekey()
  File "/usr/lib/python3.8/site-packages/OpenSSL/crypto.py", line 3177, in load_pkcs12
    _raise_current_error()
  File "/usr/lib/python3.8/site-packages/OpenSSL/_util.py", line 57, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.crypto.Error: [('asn1 encoding routines', 'ASN1_get_object', 'header too long')]
2020-11-28 23:28:04,363:ERROR:certbot._internal.log:An unexpected error occurred:
2020-11-28 23:28:04,364:ERROR:certbot._internal.log:OpenSSL.crypto.Error: [('asn1 encoding routines', 'ASN1_get_object', 'header too long')]
```



Discussion:
Hi @theorician,

What the error here suggests is that the `private_key` field within your credentials file is not a valid PEM-encoded private key.

This is a bit hard since you can't share the file, but I think it's worth verifying that the private key is indeed valid. e.g. If you install `jq`, you can check with:

    $ jq -r .private_key /path/to/certbot.json | openssl pkey -in - -inform pem -noout -check
    Key is valid

FWIW I have tried using some Google credentials lying around with `certbot/dns-google:v1.9.0` and it works okay. So I suspect strongly that something is wrong with your `certbot.json`.

Hope that helps.
@alexzorin Thank you for the quick reply. Your help is appreciated.

I took a look as you suggested and although the credentials file looked fine at a first glance, the private key was indeed mangled by the processor when it came out of the secret store. We migrated our secret store within the last certificate refresh window. I'm glad I caught this before all our certs expired. That would have sucked.

It's a good thing that this ended up being user error rather than a bug. However, can I petition for a more user-friendly error message for this?

Resolving patch:
diff --git a/certbot-dns-google/certbot_dns_google/_internal/dns_google.py b/certbot-dns-google/certbot_dns_google/_internal/dns_google.py
--- a/certbot-dns-google/certbot_dns_google/_internal/dns_google.py
+++ b/certbot-dns-google/certbot_dns_google/_internal/dns_google.py
@@ -85,9 +85,13 @@ def __init__(self, account_json=None, dns_api=None):
 
         scopes = ['https://www.googleapis.com/auth/ndev.clouddns.readwrite']
         if account_json is not None:
-            credentials = ServiceAccountCredentials.from_json_keyfile_name(account_json, scopes)
-            with open(account_json) as account:
-                self.project_id = json.load(account)['project_id']
+            try:
+                credentials = ServiceAccountCredentials.from_json_keyfile_name(account_json, scopes)
+                with open(account_json) as account:
+                    self.project_id = json.load(account)['project_id']
+            except Exception as e:
+                raise errors.PluginError(
+                    "Error parsing credentials file '{}': {}".format(account_json, e))
         else:
             credentials = None
             self.project_id = self.get_project_id()


Test patch:
diff --git a/certbot-dns-google/tests/dns_google_test.py b/certbot-dns-google/tests/dns_google_test.py
--- a/certbot-dns-google/tests/dns_google_test.py
+++ b/certbot-dns-google/tests/dns_google_test.py
@@ -107,6 +107,17 @@ def test_client_without_credentials(self, get_project_id_mock, credential_mock,
         self.assertFalse(credential_mock.called)
         self.assertTrue(get_project_id_mock.called)
 
+    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    def test_client_bad_credentials_file(self, credential_mock):
+        credential_mock.side_effect = ValueError('Some exception buried in oauth2client')
+        with self.assertRaises(errors.PluginError) as cm:
+            self._setUp_client_with_mock([])
+        self.assertEqual(
+            str(cm.exception),
+            "Error parsing credentials file '/not/a/real/path.json': "
+            "Some exception buried in oauth2client"
+        )
+
     @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)