Repository: certbot/certbot

Issue:
"Could not automatically find a matching server block." on custom nginx configuration
Using certbot for a custom reverse proxy configuration, I received the following error:

> Could not automatically find a matching server block. Set the `server_name` directive to use the Nginx installer.


## My operating system is (include version):
Debian GNU/Linux 9

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):
```bash
wget https://dl.eff.org/certbot-auto \
    && chmod a+x certbot-auto \
    && mv ./certbot-auto /usr/local/sbin/ \
```

## I ran this command and it produced this output:
`certbot-auto run -n --nginx --expand --no-redirect -d sub.private.com -d www.private2.com -d www.private3.com`

## Certbot's behavior differed from what I expected because:
This is the output:
```
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for sub.private.com
tls-sni-01 challenge for www.private2.com
http-01 challenge for www.private3.com
Cleaning up challenges
2018/04/03 13:55:06 [notice] 47#47: signal process started
Could not automatically find a matching server block. Set the `server_name` directive to use the Nginx installer.
```

## Here is a Certbot log showing the issue (if available):
```
[…]
2018-04-03 13:55:06,340:INFO:certbot.auth_handler:tls-sni-01 challenge for sub.private.com
2018-04-03 13:55:06,340:INFO:certbot.auth_handler:tls-sni-01 challenge for www.private2.com
2018-04-03 13:55:06,341:INFO:certbot.auth_handler:http-01 challenge for www.private3.com
2018-04-03 13:55:06,368:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py", line 124, in _solve_challenges
    resp = self.auth.perform(all_achalls)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 968, in perform
    sni_response = sni_doer.perform()
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/tls_sni_01.py", line 58, in perform
    vhosts = self.configurator.choose_vhosts(achall.domain, create_if_no_match=True)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 285, in choose_vhosts
    vhosts = [self._vhost_from_duplicated_default(target_name)]
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 330, in _vhost_from_duplicated_default
    default_vhost = self._get_default_vhost(port)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 361, in _get_default_vhost
    raise errors.MisconfigurationError("Could not automatically find a matching server"
MisconfigurationError: Could not automatically find a matching server block. Set the `server_name` directive to use the Nginx installer.

2018-04-03 13:55:06,368:DEBUG:certbot.error_handler:Calling registered functions
2018-04-03 13:55:06,368:INFO:certbot.auth_handler:Cleaning up challenges
2018-04-03 13:55:07,607:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/opt/eff.org/certbot/venv/bin/letsencrypt", line 11, in <module>
    sys.exit(main())
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py", line 1266, in main
    return config.func(config, plugins)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py", line 1031, in run
    certname, lineage)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py", line 113, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/renewal.py", line 297, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/client.py", line 294, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/client.py", line 330, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py", line 72, in handle_authorizations
    resp = self._solve_challenges(aauthzrs)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py", line 124, in _solve_challenges
    resp = self.auth.perform(all_achalls)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 968, in perform
    sni_response = sni_doer.perform()
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/tls_sni_01.py", line 58, in perform
    vhosts = self.configurator.choose_vhosts(achall.domain, create_if_no_match=True)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 285, in choose_vhosts
    vhosts = [self._vhost_from_duplicated_default(target_name)]
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 330, in _vhost_from_duplicated_default
    default_vhost = self._get_default_vhost(port)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 361, in _get_default_vhost
    raise errors.MisconfigurationError("Could not automatically find a matching server"
MisconfigurationError: Could not automatically find a matching server block. Set the `server_name` directive to use the Nginx installer.
```

## Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:
```
server {
    listen 80;

    server_name _;

    return 301 https://$host$request_uri;
}

##
# Custom domains
##
server {
    server_name sub.private.com www.private2.com www.private3.com;

    listen 443 ssl;
    ssl_certificate     /etc/letsencrypt/live/letsencrypt.starteed.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/letsencrypt.starteed.com/privkey.pem; # managed by Certbot
}

server {
    […]
}

server {
    […]
}
```


Discussion:
Hi @FedericoBiccheddu, thanks for posting! This bug often happens when we have trouble parsing something in the Nginx config file. The following information will help me debug your issue. If you don't feel comfortable posting any of publicly, feel free to email it to me at erica@eff.org:

- certbot version, found with `certbot-auto --version`
- the results of `tree /etc/nginx/`
- the results of `nginx -t`
- the full contents of the log you posted a snippet of above
- the full contents of the nginx config file you posted a shortened version of above
- the contents of `/etc/nginx/nginx.conf` if that's a different file from the one you posted above, or the contents of the file mentioned in `nginx -t` if the root config file is located elsewhere from `/etc/nginx/nginx.conf`
If I may chime in, I have just came across the same problem as quoted in the title.

I ran DigitalOcean's [guide](https://www.digitalocean.com/community/tutorials/how-to-set-up-let-s-encrypt-with-nginx-server-blocks-on-ubuntu-16-04) successfully in the beginning of March.

When running Step 5 of the guide just now: `certbot renew --dry-run` I received the same error as the title of this thread:
> Attempting to renew cert (mydomein.ext) from /etc/letsencrypt/renewal/mydomein.ext.conf produced an unexpected error: Could not automatically find a matching server block. Set the `server_name` directive to use the Nginx installer.. Skipping.
> All renewal attempts failed. The following certs could not be renewed:
>   /etc/letsencrypt/live/mydomein.ext/fullchain.pem (failure)

My `certbot --version` is certbot 0.21.1

My `tree /etc/nginx/` is (domain' edited)
/etc/nginx/
├── conf.d
├── fastcgi.conf
├── fastcgi_params
├── koi-utf
├── koi-win
├── mime.types
├── nginx.conf
├── proxy_params
├── scgi_params
├── sites-available
│   ├── default
│   ├── default.bak
│   └── default.bak1
├── sites-enabled
│   ├── default -> /etc/nginx/sites-available/default
│   └── domain2.com
├── snippets
│   ├── fastcgi-php.conf
│   ├── self-signed.conf
│   ├── snakeoil.conf
│   └── ssl-params.conf
├── uwsgi_params
└── win-utf

`nginx -t` output:
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Am happy to share the rest of the configs/diagnostics in private if it helps.

I would be very grateful for any pointers to resolving this issue.
Version:
`certbot 0.23.0`

Tree:
```
/etc/nginx/
├── conf.d
├── fastcgi.conf
├── fastcgi_params
├── koi-utf
├── koi-win
├── mime.types
├── nginx.conf
├── proxy_params
├── scgi_params
├── sites-available
│   ├── default
│   └── default.save
├── sites-enabled
│   └── default -> /etc/nginx/sites-available/default
├── snippets
│   ├── fastcgi-php.conf
│   └── snakeoil.conf
├── uwsgi_params
└── win-utf
```
nginx -t:
```
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
```

I have the same problem trying to expand an existing list of domains. 
```
(E)xpand/(C)ancel: E
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for xxxx.com
http-01 challenge for yyy.xxxx.com
tls-sni-01 challenge for aaa.xxxx.com
tls-sni-01 challenge for bbb.xxxx.com
tls-sni-01 challenge for ccc.xxxx.com
http-01 challenge for new.xxxx.com
Cleaning up challenges
Could not automatically find a matching server block. Set the `server_name` directive to use the Nginx installer.
```
Traceback with the verbose flag:
```
Traceback (most recent call last):
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py", line 73, in handle_authorizations
    resp = self._solve_challenges(aauthzrs)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py", line 124, in _solve_challenges
    resp = self.auth.perform(all_achalls)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 1032, in perform
    sni_response = sni_doer.perform()
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/tls_sni_01.py", line 58, in perform
    vhosts = self.configurator.choose_vhosts(achall.domain, create_if_no_match=True)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 289, in choose_vhosts
    vhosts = [self._vhost_from_duplicated_default(target_name)]
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 334, in _vhost_from_duplicated_default
    default_vhost = self._get_default_vhost(port)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 366, in _get_default_vhost
    raise errors.MisconfigurationError("Could not automatically find a matching server"
MisconfigurationError: Could not automatically find a matching server block. Set the `server_name` directive to use the Nginx installer.
```

Server block from nginx config:
```
server {
	listen 80;
	server_name new.xxxx.com;
    error_log /var/log/nginx/new_error.log;
    access_log /var/log/nginx/new_access.log;

    root /var/www/html;
    index index.html index.htm index.nginx-debian.html;

    location / {
		try_files $uri $uri/ =404;
    }
}
```
These issues are due to configurations we have trouble parsing, so seeing the entire relevant configuration files is necessary to debug. Please send relevant files to erica@eff.org mentioning this issue number in the subject line for further assistance.
One problem here (@aredey's issue) is when there are multiple default server blocks, we aren't sure which to pick, and subsequently fail. I'll fix that up. @FedericoBiccheddu and @illustris, if you send me the contents of all relevant nginx configuration files (including `nginx.conf`, and anything in `sites-available` or `conf.d`) I can check if this is also your issue.

Resolving patch:
diff --git a/certbot-nginx/certbot_nginx/configurator.py b/certbot-nginx/certbot_nginx/configurator.py
--- a/certbot-nginx/certbot_nginx/configurator.py
+++ b/certbot-nginx/certbot_nginx/configurator.py
@@ -289,7 +289,8 @@ def choose_vhosts(self, target_name, create_if_no_match=False):
         if not vhosts:
             if create_if_no_match:
                 # result will not be [None] because it errors on failure
-                vhosts = [self._vhost_from_duplicated_default(target_name)]
+                vhosts = [self._vhost_from_duplicated_default(target_name, True,
+                    str(self.config.tls_sni_01_port))]
             else:
                 # No matches. Raise a misconfiguration error.
                 raise errors.MisconfigurationError(
@@ -332,9 +333,12 @@ def ipv6_info(self, port):
                     ipv6only_present = True
         return (ipv6_active, ipv6only_present)
 
-    def _vhost_from_duplicated_default(self, domain, port=None):
+    def _vhost_from_duplicated_default(self, domain, allow_port_mismatch, port):
+        """if allow_port_mismatch is False, only server blocks with matching ports will be
+           used as a default server block template.
+        """
         if self.new_vhost is None:
-            default_vhost = self._get_default_vhost(port, domain)
+            default_vhost = self._get_default_vhost(domain, allow_port_mismatch, port)
             self.new_vhost = self.parser.duplicate_vhost(default_vhost,
                 remove_singleton_listen_params=True)
             self.new_vhost.names = set()
@@ -350,19 +354,24 @@ def _add_server_name_to_vhost(self, vhost, domain):
             name_block[0].append(name)
         self.parser.update_or_add_server_directives(vhost, name_block)
 
-    def _get_default_vhost(self, port, domain):
+    def _get_default_vhost(self, domain, allow_port_mismatch, port):
+        """Helper method for _vhost_from_duplicated_default; see argument documentation there"""
         vhost_list = self.parser.get_vhosts()
         # if one has default_server set, return that one
-        default_vhosts = []
+        all_default_vhosts = []
+        port_matching_vhosts = []
         for vhost in vhost_list:
             for addr in vhost.addrs:
                 if addr.default:
-                    if port is None or self._port_matches(port, addr.get_port()):
-                        default_vhosts.append(vhost)
-                        break
+                    all_default_vhosts.append(vhost)
+                    if self._port_matches(port, addr.get_port()):
+                        port_matching_vhosts.append(vhost)
+                    break
 
-        if len(default_vhosts) == 1:
-            return default_vhosts[0]
+        if len(port_matching_vhosts) == 1:
+            return port_matching_vhosts[0]
+        elif len(all_default_vhosts) == 1 and allow_port_mismatch:
+            return all_default_vhosts[0]
 
         # TODO: present a list of vhosts for user to choose from
 
@@ -471,7 +480,7 @@ def choose_redirect_vhosts(self, target_name, port, create_if_no_match=False):
             matches = self._get_redirect_ranked_matches(target_name, port)
             vhosts = [x for x in [self._select_best_name_match(matches)]if x is not None]
         if not vhosts and create_if_no_match:
-            vhosts = [self._vhost_from_duplicated_default(target_name, port=port)]
+            vhosts = [self._vhost_from_duplicated_default(target_name, False, port)]
         return vhosts
 
     def _port_matches(self, test_port, matching_port):


Test patch:
diff --git a/certbot-nginx/certbot_nginx/tests/configurator_test.py b/certbot-nginx/certbot_nginx/tests/configurator_test.py
--- a/certbot-nginx/certbot_nginx/tests/configurator_test.py
+++ b/certbot-nginx/certbot_nginx/tests/configurator_test.py
@@ -722,6 +722,13 @@ def test_deploy_no_match_fail_multiple_defaults(self):
             "www.nomatch.com", "example/cert.pem", "example/key.pem",
             "example/chain.pem", "example/fullchain.pem")
 
+    def test_deploy_no_match_multiple_defaults_ok(self):
+        foo_conf = self.config.parser.abs_path('foo.conf')
+        self.config.parser.parsed[foo_conf][2][1][0][1][0][1] = '*:5001'
+        self.config.version = (1, 3, 1)
+        self.config.deploy_cert("www.nomatch.com", "example/cert.pem", "example/key.pem",
+            "example/chain.pem", "example/fullchain.pem")
+
     def test_deploy_no_match_add_redirect(self):
         default_conf = self.config.parser.abs_path('sites-enabled/default')
         foo_conf = self.config.parser.abs_path('foo.conf')