Repository: Mailu/Mailu

Issue:
mail-letsencrypt -> front --> Invalid response from http://mail.example.fr/
## Environment & Version

### Environment

- [*] docker compose

### Version

- Version: `master` 2.0

## Description

After setting up Mailu behind a reverse proxy with the mail-letsencrypt TLS setting, I cannot renew my certificates.

After checking the logs for the front container, I get Error: 
```
Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
  Domain: mail.example.fr
  Type:   unauthorized
  Detail: XXX.XXX.XXX.XXX: Invalid response from http://mail.example.fr/.well-known/acme-challenge/jw_V7ZveSz5gp2Gypw6et4LczGynp8veedfJ5f4-AF4: 404
```
## Replication Steps

1. Rebuild Mailu using the docker-compose settings below
2. Try to add an account in my client => Certificate is expired

## Expected behaviour

Certificate renewal

## Logs

```
2023-10-06 13:55:02,423:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 06 Oct 2023 13:55:02 GMT
Content-Type: application/json
Content-Length: 1026
Connection: keep-alive
Boulder-Requester: 405748950
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 3hclikJOYQeRqfKi-jIPScoJJR3C-lh4lt7atk0b2yydJ2qurDI
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "mail.example.fr"
  },
  "status": "invalid",
  "expires": "2023-10-13T13:54:59Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "XXX.XXX.XXX.XXX: Invalid response from http://mail.example.fr/.well-known/acme-challenge/jw_V7ZveSz5gp2Gypw6et4LczGynp8veedfJ5f4-AF4: 404",
        "status": 403
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/271292522506/NQsvZw",
      "token": "jw_V7ZveSz5gp2Gypw6et4LczGynp8veedfJ5f4-AF4",
      "validationRecord": [
        {
          "url": "http://mail.example.fr/.well-known/acme-challenge/jw_V7ZveSz5gp2Gypw6et4LczGynp8veedfJ5f4-AF4",
          "hostname": "mail.example.fr",
          "port": "80",
          "addressesResolved": [
            "XXX.XXX.XXX.XXX"
          ],
          "addressUsed": "XXX.XXX.XXX.XXX"
        }
      ],
      "validated": "2023-10-06T13:55:00Z"
    }
  ]
}
2023-10-06 13:55:02,424:DEBUG:acme.client:Storing nonce: 3hclikJOYQeRqfKi-jIPScoJJR3C-lh4lt7atk0b2yydJ2qurDI
2023-10-06 13:55:02,424:INFO:certbot._internal.auth_handler:Challenge failed for domain mail.example.fr
```

## Configuration files

### mailu.env:

```
# Mailu main configuration file
#
# This file is autogenerated by the configuration management wizard for compose flavor.
# For a detailed list of configuration variables, see the documentation at
# https://mailu.io

###################################
# Common configuration variables
###################################

# Set to a randomly generated 16 bytes string
SECRET_KEY=REDACTED4PRIVACY

# Subnet of the docker network. This should not conflict with any networks to which your system is connected. (Internal and external!)
SUBNET=192.168.203.0/24

# Main mail domain
DOMAIN=2rock.fr

# Hostnames for this server, separated with comas
HOSTNAMES=mail.example.fr,autoconfig.example.fr,autoconfig.example.fr

# Postmaster local part (will append the main mail domain)
POSTMASTER=admin

# Choose how secure connections will behave (value: letsencrypt, cert, notls, mail, mail-letsencrypt)
TLS_FLAVOR=mail-letsencrypt

# Authentication rate limit per IP (per /24 on ipv4 and /48 on ipv6)
AUTH_RATELIMIT_IP=5/hour

# Authentication rate limit per user (regardless of the source-IP)
AUTH_RATELIMIT_USER=10/day

#
AUTH_RATELIMIT_EXEMPTION_LENGTH=86400

# Opt-out of statistics, replace with "True" to opt out
DISABLE_STATISTICS=True

###################################
# Optional features
###################################

# Expose the admin interface (value: true, false)
ADMIN=true

# Choose which webmail to run if any (values: roundcube, snappymail, none)
WEBMAIL=roundcube

# Expose the API interface (value: true, false)
API=true

# Dav server implementation (value: radicale, none)
WEBDAV=radicale

# Antivirus solution (value: clamav, none)
ANTIVIRUS=clamav

# Scan Macros solution (value: true, false)
SCAN_MACROS=true

###################################
# Mail settings
###################################

# Message size limit in bytes
# Default: accept messages up to 50MB
# Max attachment size will be 33% smaller
MESSAGE_SIZE_LIMIT=50000000

# Message rate limit (per user)
MESSAGE_RATELIMIT=200/day

# Networks granted relay permissions
# Use this with care, all hosts in this networks will be able to send mail without authentication!
RELAYNETS=

# Will relay all outgoing mails if configured
RELAYHOST=

# Enable fetchmail
FETCHMAIL_ENABLED=true

# Fetchmail delay
FETCHMAIL_DELAY=600

# Recipient delimiter, character used to delimiter localpart from custom address part
RECIPIENT_DELIMITER=+

# DMARC rua and ruf email
DMARC_RUA=admin
DMARC_RUF=admin

# Welcome email, enable and set a topic and body if you wish to send welcome
# emails to all users.
WELCOME=false
WELCOME_SUBJECT=Welcome to your new email account
WELCOME_BODY=Welcome to your new email account, if you can read this, then it is configured properly!

# Maildir Compression
# choose compression-method, default: none (value: gz, bz2, zstd)
COMPRESSION=bz2
# change compression-level, default: 6 (value: 1-9)
COMPRESSION_LEVEL=6

# IMAP full-text search is enabled by default. Set the following variable to off in order to disable the feature.
# FULL_TEXT_SEARCH=off

###################################
# Web settings
###################################

# Path to redirect / to
WEBROOT_REDIRECT=/webmail

# Path to the admin interface if enabled
WEB_ADMIN=/admin

# Path to the webmail if enabled
WEB_WEBMAIL=/webmail

# Path to the API interface if enabled
WEB_API=/api

# Website name
SITENAME=Mailu

# Linked Website URL
WEBSITE=https://example.Fr



###################################
# Advanced settings
###################################

# Docker-compose project name, this will prepended to containers names.
COMPOSE_PROJECT_NAME=mailu

# Number of rounds used by the password hashing scheme
CREDENTIAL_ROUNDS=12

# Header to take the real ip from
REAL_IP_HEADER=X-Real-IP

# IPs for nginx set_real_ip_from (CIDR list separated by commas)
REAL_IP_FROM=10.0.0.3

# choose wether mailu bounces (no) or rejects (yes) mail when recipient is unknown (value: yes, no)
REJECT_UNLISTED_RECIPIENT=

# Log level threshold in start.py (value: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET)
LOG_LEVEL=INFO

# Timezone for the Mailu containers. See this link for all possible values https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
TZ=Etc/UTC

# Default spam threshold used for new users
DEFAULT_SPAM_THRESHOLD=80

# API token required for authenticating to the RESTful API.
# This is a mandatory setting for using the RESTful API.
API_TOKEN=REDACTED4PRIVACY
```

### docker-compose.yml:

```
*# This file is auto-generated by the Mailu configuration wizard.
# Please read the documentation before attempting any change.
# Generated for compose flavor

version: '2.2'

services:

  # External dependencies
  redis:
    image: redis:alpine
    restart: always
    volumes:
      - "/opt/mailu/redis:/data"
    depends_on:
      - resolver
    dns:
      - 192.168.203.254

  # Core services
  front:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-2.0}
    restart: always
    env_file: mailu.env
    logging:
      driver: journald
      options:
        tag: mailu-front
    ports:
      - "10.0.0.5:80:80"
      - "10.0.0.5:443:443"
      - "10.0.0.5:25:25"
      - "10.0.0.5:465:465"
      - "10.0.0.5:587:587"
      - "10.0.0.5:110:110"
      - "10.0.0.5:995:995"
      - "10.0.0.5:143:143"
      - "10.0.0.5:993:993"
    networks:
      - default
      - webmail
      - radicale
    volumes:
      - "/opt/mailu/certs:/certs"
      - "/opt/mailu/overrides/nginx:/overrides:ro"
    depends_on:
      - resolver
    dns:
      - 192.168.203.254

  resolver:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}unbound:${MAILU_VERSION:-2.0}
    env_file: mailu.env
    restart: always
    networks:
      default:
        ipv4_address: 192.168.203.254

  admin:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-2.0}
    restart: always
    env_file: mailu.env
    logging:
      driver: journald
      options:
        tag: mailu-admin
    volumes:
      - "/opt/mailu/data:/data"
      - "/opt/mailu/dkim:/dkim"
    depends_on:
      - redis
      - resolver
    dns:
      - 192.168.203.254

  imap:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}dovecot:${MAILU_VERSION:-2.0}
    restart: always
    env_file: mailu.env
    logging:
      driver: journald
      options:
        tag: mailu-imap
    volumes:
      - "/opt/mailu/mail:/mail"
      - "/opt/mailu/overrides/dovecot:/overrides:ro"
    depends_on:
      - front
      - resolver
    dns:
      - 192.168.203.254

  smtp:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}postfix:${MAILU_VERSION:-2.0}
    restart: always
    env_file: mailu.env
    logging:
      driver: journald
      options:
        tag: mailu-smtp
    volumes:
      - "/opt/mailu/mailqueue:/queue"
      - "/opt/mailu/overrides/postfix:/overrides:ro"
    depends_on:
      - front
      - resolver
    dns:
      - 192.168.203.254

  oletools:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}oletools:${MAILU_VERSION:-2.0}
    hostname: oletools
    restart: always
    networks:
      - noinet
    depends_on:
      - resolver
    dns:
      - 192.168.203.254

  antispam:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}rspamd:${MAILU_VERSION:-2.0}
    hostname: antispam
    restart: always
    env_file: mailu.env
    logging:
      driver: journald
      options:
        tag: mailu-antispam
    networks:
      - default
      - noinet
    volumes:
      - "/opt/mailu/filter:/var/lib/rspamd"
      - "/opt/mailu/overrides/rspamd:/overrides:ro"
    depends_on:
      - front
      - redis
      - oletools
      - antivirus
      - resolver
    dns:
      - 192.168.203.254

  # Optional services
  antivirus:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}clamav:${MAILU_VERSION:-2.0}
    restart: always
    env_file: mailu.env
    volumes:
      - "/opt/mailu/filter:/data"
    depends_on:
      - resolver
    dns:
      - 192.168.203.254

  webdav:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}radicale:${MAILU_VERSION:-2.0}
    restart: always
    volumes:
      - "/opt/mailu/dav:/data"
    networks:
      - radicale

  fetchmail:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}fetchmail:${MAILU_VERSION:-2.0}
    restart: always
    env_file: mailu.env
    volumes:
      - "/opt/mailu/data/fetchmail:/data"
    depends_on:
      - admin
      - smtp
      - imap
      - resolver
    dns:
      - 192.168.203.254

  # Webmail
  webmail:
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}webmail:${MAILU_VERSION:-2.0}
    restart: always
    env_file: mailu.env
    volumes:
      - "/opt/mailu/webmail:/data"
      - "/opt/mailu/overrides/roundcube:/overrides:ro"
    networks:
      - webmail
    depends_on:
      - front

networks:
  default:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 192.168.203.0/24
  radicale:
    driver: bridge
  webmail:
    driver: bridge
  noinet:
    driver: bridge
    internal: true
```

### nginx reverse proxy configuration mail.example.fr.conf

```
server {
        server_name mail.example.fr;

        listen 80;
        listen [::]:80;

        location ^~ /.well-known/acme-challenge/ {
                #alias /var/www/html/mail.example.fr/.well-know;
                proxy_pass http://10.0.0.5/.well-known/acme-challenge/;
                proxy_set_header Host $host;
        }

        location / {
                return 301 https://$host$request_uri;
        }
}

server {
        server_name mail.example.fr;

        listen 443 ssl http2;
        listen [::]:443 ssl http2;

        location / {
                rewrite / https://mail.example.fr/webmail redirect;
        }

        location ~* ^/(admin|api|sso|static|webdav|webmail|(apple\.)?mobileconfig|(\.well\-known/autoconfig/)?mail/|Autodiscover/Autodiscover) {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass https://10.0.0.5;
        }

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_headers_hash_max_size 512;
        proxy_headers_hash_bucket_size 128;

        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload";
        client_max_body_size 0;

        ssl_certificate /etc/letsencrypt/live/mail.example.fr/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/mail.example.fr/privkey.pem;
        ssl_trusted_certificate /etc/letsencrypt/live/mail.example.fr/chain.pem;

        include /etc/nginx/snippets/header.conf;
        include /etc/nginx/snippets/ssl.conf;

        access_log /var/log/nginx/mail.example.fr.access.log;
        error_log /var/log/nginx/mail.example.fr.error.log;
}
```
### DNS configuration

![Sans titre-1](https://github.com/Mailu/Mailu/assets/2854830/b05b8875-630e-4552-afc0-d12a683bda09)



Discussion:
It's most likely your reverse proxy's configuration.

You have a single ``server_name`` defined for each block whereas you have multiple in ``HOSTNAMES`` and ``DOMAIN``. Add the missing ones and it should start working.

If it doesn't, use tcpdump and try to find out how far the request on port 80 goes: is it going past your reverse proxy? is it going past front and to port 8008 like it ought to?

Resolving patch:
diff --git a/core/admin/start.py b/core/admin/start.py
--- a/core/admin/start.py
+++ b/core/admin/start.py
@@ -29,7 +29,7 @@ def test_unsupported():
     import codecs
     if os.path.isfile(codecs.decode('/.qbpxrerai', 'rot13')) or os.environ.get(codecs.decode('V_XABJ_ZL_FRGHC_QBRFAG_SVG_ERDHVERZRAGF_NAQ_JBAG_SVYR_VFFHRF_JVGUBHG_CNGPURF', 'rot13'), None):
         return
-    print('Your system is not supported. Please start by reading the documentation and then http://www.catb.org/~esr/faqs/smart-questions.html')
+    log.critical('Your system is not supported. Please start by reading the documentation and then http://www.catb.org/~esr/faqs/smart-questions.html')
     while True:
         time.sleep(5)
 
diff --git a/core/nginx/letsencrypt.py b/core/nginx/letsencrypt.py
--- a/core/nginx/letsencrypt.py
+++ b/core/nginx/letsencrypt.py
@@ -61,13 +61,13 @@ def serve_one_request():
 # Run certbot every day
 while True:
     while True:
-        hostname = os.environ['HOSTNAMES'].split(' ')[0]
+        hostname = os.environ['HOSTNAMES'].split(',')[0]
         target = f'http://{hostname}/.well-known/acme-challenge/testing'
         thread = Thread(target=serve_one_request)
         thread.start()
         r = requests.get(target)
         if r.status_code != 204:
-            log.error(f"Can't reach {target}!, please ensure it's fixed or change the TLS_FLAVOR.")
+            log.critical(f"Can't reach {target}!, please ensure it's fixed or change the TLS_FLAVOR.")
             time.sleep(5)
         else:
             break


Test patch: