Repository: ansible/ansible

Issue:
Ansible 2.4.0 letsencrypt module does not properly request a challenge, in DNS-01 mode
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
- letsencrypt

##### ANSIBLE VERSION
```
ansible 2.4.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/ajacocks/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.13 (default, Sep  5 2017, 08:53:59) [GCC 7.1.1 20170622 (Red Hat 7.1.1-3)]
```

##### CONFIGURATION
```bash
$ ansible-config dump --only-changed
$
```

##### OS / ENVIRONMENT
- Fedora 26 x86_64, patched to current
- only letsencrypt module, attempting to generate certs against the staging environment

##### SUMMARY
- Ansible 2.4.0 letsencrypt module does not properly request a challenge, in DNS-01 mode

##### STEPS TO REPRODUCE
```yaml
$ cat playbook.yml
---
- name: Generate Let's Encrypt certificates
  hosts: localhost
  become: yes
  vars:
  - ler53_account_email: "alexander@redhat.com"
  - ler53_cert_key_usages:
    - digitalSignature
    - keyEncipherment
  - ler53_cert_common_name: "host.0.domain"
  - ler53_cert_sans:
    - "DNS:host.0.domain"
    - "DNS:host.1.domain"
  roles:
    - ler53
...
$ cat roles/ler53/tasks/main.yml
---
- name: install required packages
  package:
    name: "{{ item }}"
    state: present
  with_items:
  - openssl
  - python-boto
  - pyOpenSSL

- name: "create the {{ ler53_cert_dir }} directory"
  file:
    path: "{{ ler53_cert_dir }}"
    state: directory
    owner: root
    mode: 0755

- name: generate the private key
  openssl_privatekey:
    path: "{{ ler53_cert_dir + '/' + ler53_key_file_name }}"
    size: "{{ ler53_key_size }}"

- name: set the private key file permissions
  file:
    path: "{{ ler53_cert_dir + '/' + ler53_key_file_name }}"
    owner: "{{ ler53_cert_files_owner }}"
    group: "{{ ler53_cert_files_group }}"
    mode: "{{ ler53_cert_files_mode }}"

- name: generate the CSR
  openssl_csr:
    email_address: "{{ ler53_account_email }}"
    key_usage: "{{ ler53_cert_key_usages }}"
    path: "{{ ler53_cert_dir + '/' + ler53_csr_file_name }}"
    privatekey_path: "{{ ler53_cert_dir + '/' + ler53_key_file_name }}"
    subject_alt_name: "{{ ler53_cert_sans  }}"
    subject_alt_name_critical: true
  register: csr_results

- name: set the CSR file permissions
  file:
    path: "{{ ler53_cert_dir + '/' + ler53_csr_file_name }}"
    owner: "{{ ler53_cert_files_owner }}"
    group: "{{ ler53_cert_files_group }}"
    mode: "{{ ler53_cert_files_mode }}"

- name: "create the {{ ler53_account_key_dir }} directory"
  file:
    path: "{{ ler53_account_key_dir }}"
    owner: root
    mode: 0700
    state: directory

- name: generate the Let's Encrypt account key
  openssl_privatekey:
    path: "{{ ler53_account_key_dir + '/' + ler53_account_key_file_name }}"
    size: "{{ ler53_key_size }}"

- name: set the Let's Encrypt account key file permissions
  file:
    path: "{{ ler53_account_key_dir + '/' + ler53_account_key_file_name }}"
    owner: root
    mode: 0600

- name: initiate the Let's Encrypt challenge
  letsencrypt:
    acme_directory: "{{ ler53_acme_directory }}"
    challenge: "dns-01"
    account_key: "{{ ler53_account_key_dir + '/' + ler53_account_key_file_name }}"
    csr: "{{ ler53_cert_dir + '/' + ler53_csr_file_name }}"
    dest: "{{ ler53_cert_dir + '/' + ler53_cert_file_name }}"
    account_email: "{{ ler53_account_email }}"
    remaining_days: "{{ ler53_cert_remaining_days_before_renewal }}"
  register: lets_encrypt_challenge
  ignore_errors: yes

- debug:
    var: lets_encrypt_challenge
...
```

##### EXPECTED RESULTS
The "lets_encrypt_challenge" variable should have contained challenge data to be loaded into my domain's TXT records.

##### ACTUAL RESULTS
```bash
$ ansible-playbook playbook.yml -K -vvv
<trimmed>
TASK [ler53 : initiate the Let's Encrypt challenge] ****************************
task path: /home/ajacocks/src/le-aws/roles/ler53/tasks/main.yml:92
Using module file /usr/lib/python2.7/site-packages/ansible/modules/web_infrastructure/letsencrypt.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: ajacocks
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/ajacocks/.ansible/tmp/ansible-tmp-1510261206.42-149107025083243 `" && echo ansible-tmp-1510261206.42-149107025083243="` echo /home/ajacocks/.ansible/tmp/ansible-tmp-1510261206.42-149107025083243 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmpytd0c_ TO /home/ajacocks/.ansible/tmp/ansible-tmp-1510261206.42-149107025083243/letsencrypt.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/ajacocks/.ansible/tmp/ansible-tmp-1510261206.42-149107025083243/ /home/ajacocks/.ansible/tmp/ansible-tmp-1510261206.42-149107025083243/letsencrypt.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=aeqsrezayvbvzoncjpafaoqfyqdqtfvo] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-aeqsrezayvbvzoncjpafaoqfyqdqtfvo; /usr/bin/python2 /home/ajacocks/.ansible/tmp/ansible-tmp-1510261206.42-149107025083243/letsencrypt.py; rm -rf "/home/ajacocks/.ansible/tmp/ansible-tmp-1510261206.42-149107025083243/" > /dev/null 2>&1'"'"' && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "account_email": "alexander@redhat.com", 
            "account_key": "/etc/ssl/lets_encrypt/lets_encrypt_account.key", 
            "acme_directory": "https://acme-staging.api.letsencrypt.org/directory", 
            "agreement": "https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf", 
            "challenge": "dns-01", 
            "csr": "/etc/ssl/host.0.domain/host.0.domain.csr", 
            "data": null, 
            "dest": "/etc/ssl/host.0.domain/host.0.domain.crt", 
            "remaining_days": 10
        }
    }, 
    "msg": "Error new cert: CODE: 403 RESULT: {\n  \"type\": \"urn:acme:error:unauthorized\",\n  \"detail\": \"Error creating new cert :: authorizations for these names not found or expired: host.0.domain, host.1.domain\",\n  \"status\": 403\n}"
}
...ignoring

TASK [ler53 : debug] ***********************************************************
task path: /home/ajacocks/src/le-aws/roles/ler53/tasks/main.yml:104
ok: [localhost] => {
    "lets_encrypt_challenge": {
        "changed": false, 
        "failed": true, 
        "msg": "Error new cert: CODE: 403 RESULT: {\n  \"type\": \"urn:acme:error:unauthorized\",\n  \"detail\": \"Error creating new cert :: authorizations for these names not found or expired: host.0.domain, host.1.domain\",\n  \"status\": 403\n}"
    }
}
```


Discussion:
cc @mgruener
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I'm seeing the same in Ansible 2.2.1 (Debian package). 

Setting the "challenge" on the first invocation to "http-01" doesn't help.

@ajacocks : did this work before for you? Because I think it did for me. LetsEncrypt changed the server side 3 hours before you opened this issue (https://twitter.com/letsencrypt_ops/status/928692468169576448), which may be related.
@cmock Unfortuntely, I hadn't tried this before, as I had just used certbot.  That's interesting that using the HTTP challenge didn't help; I can't use that, in my case, so I didn't try.

Can you post a log of your attempt, using HTTP-01?  I'll change the bug title to be more generic, when you do.  That may well help us to troubleshoot.

I hadn't noticed that LE changed the back end...that may well be the issue.
I cannot use http-01 for that certificate either, but the problem is in the first invocation of the letsencrypt module, which should then just tell me how to answer that challenge, right? That's why I tried http-01 as well.

I have issued and renewed a handful of certs with that ansible role before, both with http-01 and dns-01, and yesterday I've also been able to renew one of those. The problem occured for me when I tried to issue a new cert for a new CN with a new account key.

Anyways, here's the log (with all the SSH verbosity elided):

```
TASK [letsencrypt-cert : make request] *****************************************
task path: /home/cm/ansible/roles/letsencrypt-cert/tasks/main.yml:12
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/extras/web_infrastructure/letsencrypt.py
<mqtt> ESTABLISH SSH CONNECTION FOR USER: ansible
[...]
fatal: [mqtt]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "account_email": null, 
            "account_key": "/etc/ssl/private/letsencrypt-admin-cm.key", 
            "acme_directory": "https://acme-v01.api.letsencrypt.org/directory", 
            "agreement": "https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf", 
            "challenge": "http-01", 
            "csr": "/etc/ssl/my_certs/mqtt.tahina.priv.at_req.pem", 
            "data": null, 
            "dest": "/etc/ssl/my_certs/mqtt.tahina.priv.at_cert.pem", 
            "remaining_days": 15
        }, 
        "module_name": "letsencrypt"
    }, 
    "msg": "Error new cert: CODE: 403 RESULT: {\n  \"type\": \"urn:acme:error:unauthorized\",\n  \"detail\": \"Error creating new cert :: authorizations for these names not found or expired: mqtt.tahina.priv.at\",\n  \"status\": 403\n}"
}
```
Is there any way that I can help to troubleshoot this?  Is there more data that I can provide?
I encountered the exact same issue and was able to reproduce it 100% of the time when generating challenges for fqdns that were never successfully challenged through the acme protocol *and* specifying that the subjectAltName extension be critical in your CSR (which you also did in your example above).

The letsencrypt module does then not seem to be outputing challenge data for any SAN, but the common name!
I haven't investigated in the module code yet, but I thought I should give those pointers.
cc @resmo
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@oboukili I don't think that's the issue (or "openssl req" is lying about my request) -- I've included no SAN in my request, just the name in the subject.
@oboukili Also, with my example, you can see that I included both SAN and
CN.

On Sun, Nov 26, 2017 at 12:28 PM christian mock <notifications@github.com>
wrote:

> @oboukili <https://github.com/oboukili> I don't think that's the issue
> (or "openssl req" is lying about my request) -- I've included no SAN in my
> request, just the name in the subject.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/32767#issuecomment-347024559>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AVdXcjeB6-mg0q6vL-a4gdW0yU7WgaThks5s6Z_DgaJpZM4QYrok>
> .
>
-- 
=============================================================
J. Alexander Jacocks     | Senior Solutions Architect
(e) alexander@redhat.com | Public Sector Business Development
(c) (240) 447-5974       | Tyson's Corner, VA
(g) CC28 8136 C253 907A B9BB  9EEA D3C8 1A3A 790A 0AB9

@oboukili actually, you pointed us in the right direction, and it works for me now.

What I did was to add a subjectAlternateName section to the request (with "DNS:hostname"), so I supposed @ajacocks you should try to get rid of the `subject_alt_name_critical: true` when generating the CSR.

(I also switched from SHA1 to SHA256 for the CSR, but that alone wasn't the solution).
There's something fishy with the module output, I'll take a look later tomorrow maybe.
@cmock @oboukili I confirm that the `subject_alt_name_critical: true` was the problem that was blocking me.

Any thoughts on the module output?

Resolving patch:
diff --git a/lib/ansible/modules/web_infrastructure/letsencrypt.py b/lib/ansible/modules/web_infrastructure/letsencrypt.py
--- a/lib/ansible/modules/web_infrastructure/letsencrypt.py
+++ b/lib/ansible/modules/web_infrastructure/letsencrypt.py
@@ -676,7 +676,9 @@ def _get_csr_domains(self):
         common_name = re.search(r"Subject:.*? CN\s?=\s?([^\s,;/]+)", to_text(out, errors='surrogate_or_strict'))
         if common_name is not None:
             domains.add(common_name.group(1))
-        subject_alt_names = re.search(r"X509v3 Subject Alternative Name: \n +([^\n]+)\n", to_text(out, errors='surrogate_or_strict'), re.MULTILINE | re.DOTALL)
+        subject_alt_names = re.search(
+            r"X509v3 Subject Alternative Name: (?:critical)?\n +([^\n]+)\n",
+            to_text(out, errors='surrogate_or_strict'), re.MULTILINE | re.DOTALL)
         if subject_alt_names is not None:
             for san in subject_alt_names.group(1).split(", "):
                 if san.startswith("DNS:"):


Test patch: