Repository: certbot/certbot

Issue:
apache parser searching sites-available, should be sites-enabled
for obvious reasons..



Discussion:
Right now, the parser searches both... sites-enabled for what configuration directives are actually enabled and sites-available for all of the possible sites that you may have configured, for VirtualHost selection.

My idea was that Apache can automatically enable a site that may have been created, but not yet deployed yet.  If Apache is configured appropriately for Debian, sites-enabled should be a subset of sites-available.

The Apache VirtualHost objects have an attribute for if the site is currently enabled.

I do think that it makes sense to completely "generalize" the configuration by only searching sites-enabled (it would make porting to other layouts easier).

I would suggest that we don't make a distinction between sites-enabled and any other Included directory.  We should simply look through all of the current configuration files and look for VirtualHost blocks.

ah, so this is relying on sites-enabled being symlinks to sites-available then?

@jdkasten so the only thing that would break if we remove sites-available from the search path is enablement of vhosts that the admin has set up but not actually deployed?  In general that sounds slightly dangerous anyway, so if this is a simple fix we should just remove `sites-available` from whatever calls construct the search path.

@BillKeenan interested in making a PR?

I am. on it.

does anyone have some more fullsome docs on getting this running with vagrant.. currently getting this

```
Error: unauthorized :: The client lacks sufficient authorization :: Error creating new authz :: Name is blacklisted

```

after issuing this; sudo ./venv/bin/letsencrypt -d www.example.com auth

`www.example.com` is blacklisted, use real domain

this prevents me from using letsencrypt, i have no files in sites-available and the letsencrypt client tells me that no config was found and closes.

you can use the manual authorization mode until this is resolved., i did, worked well/easy.

We should definitely fix this!  @BillKeenan are you still interested in writing this PR? If not, we should assign someone the ticket...

right! Yes! I have it mostly in the can, I'll get on it tonight.

@BillKeenan nudge :)

hmm, i just ran it, and it's pulling a site from my sites-enabled without issue...

@BillKeenan do you mean you can no longer reproduce this bug, or that you have a branch that fixes it?  I can definitely still reproduce...

Can you provide steps to reproduce?

Bill Keenan
VP Technology
416.583.3772
bill@authenticweb.com

On 4 Jan 2016, at 10:37, Peter Eckersley wrote:

> @BillKeenan do you mean you can no longer reproduce this bug, or that 
> you have a branch that fixes it?  I can definitely still reproduce...
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/letsencrypt/letsencrypt/issues/1043#issuecomment-168708754

Sure!  Though I think @joohoi might have started working on this, so it may not be worth spending more time on at the moment. 

```
pde@damoclid:~/letsencrypt/letsencrypt$ ls /etc/apache2/sites-*
/etc/apache2/sites-available:

/etc/apache2/sites-enabled:
001-dee.conf  001-tweedle.conf
pde@damoclid:~/letsencrypt/letsencrypt$ cat /etc/apache2/sites-enabled/*
<VirtualHost *:80>
    ServerAdmin webmaster@localhost
        ServerName dee.isnot.org

    DocumentRoot /var/www/tweedle
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /var/www/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>

    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Order allow,deny
        Allow from all
    </Directory>

    <If "%{REQUEST_URI} =~ /thing/ || %{QUERY_STRING} == 'edit=1'">
        RewriteEngine on
        AuthType basic
        AuthBasicProvider file
        AuthUserFile "/etc/apache2/passwords"
        Require valid-user
    </If>


    ErrorLog ${APACHE_LOG_DIR}/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
<VirtualHost *:80>
    ServerAdmin webmaster@localhost
        ServerName tweedle.isnot.org

    DocumentRoot /var/www/tweedle
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /var/www/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>

    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Order allow,deny
        Allow from all
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
pde@damoclid:~/letsencrypt/letsencrypt$ sudo venv/bin/letsencrypt --apache -t

-------------------------------------------------------------------------------
No names were found in your configuration files.
You should specify ServerNames in your config files in order to allow for
accurate installation of your certificate.
If you do use the default vhost, you may specify the name manually. Would you
like to continue?
-------------------------------------------------------------------------------
(Y)es/(N)o:
```

Flagging the ticket as joohoi's for the time being.

This should have worked already, but a bug in augeas lib caused the path not to be parsed in some configurations. #2092 fixes that.

More detailed explanation:

This should have worked already - sites-enabled should be parsed because it's "Include":d from apache2.conf in the root directory, and if for some reason it was not, it can be parsed by providing --apache-vhost-root /etc/apache2/sites-enabled in the command line, provided by #1811. 

However in some cases (and especially if defined in the command line) it was NOT being parsed, because of libaugeas bug, that prevents parsing of overlapping match definitions. #2092 fixes that problem, by handling the augeas parsing paths if they overlap. Leaving only the "wildest card" match rule intact

So for now, sites-enabled can be parsed by providing cli parameters --apache-vhost-root /etc/apache2/sites-enabled and --apache-handle-sites False

--apache-vhost-root sets the vhost root to sites-enabled, and --apache-handle-sites disables the a2ensite functionality, as if this is needed, the user doesn't use the sites-available / sites-enabled & a2ensite / a2dissite configuration scheme to begin with.

This has been partially addressed, but a full fix will have to wait for 0.3.0.


Resolving patch:
diff --git a/letsencrypt-apache/letsencrypt_apache/configurator.py b/letsencrypt-apache/letsencrypt_apache/configurator.py
--- a/letsencrypt-apache/letsencrypt_apache/configurator.py
+++ b/letsencrypt-apache/letsencrypt_apache/configurator.py
@@ -488,15 +488,27 @@ def get_virtual_hosts(self):
         :rtype: list
 
         """
-        # Search vhost-root, httpd.conf for possible virtual hosts
-        paths = self.aug.match(
-            ("/files%s//*[label()=~regexp('%s')]" %
-             (self.conf("vhost-root"), parser.case_i("VirtualHost"))))
-
+        # Search base config, and all included paths for VirtualHosts
         vhs = []
-
-        for path in paths:
-            vhs.append(self._create_vhost(path))
+        vhost_paths = {}
+        for vhost_path in self.parser.parser_paths.keys():
+            paths = self.aug.match(
+                ("/files%s//*[label()=~regexp('%s')]" %
+                    (vhost_path, parser.case_i("VirtualHost"))))
+            for path in paths:
+                new_vhost = self._create_vhost(path)
+                realpath = os.path.realpath(new_vhost.filep)
+                if realpath not in vhost_paths.keys():
+                    vhs.append(new_vhost)
+                    vhost_paths[realpath] = new_vhost.filep
+                elif realpath == new_vhost.filep:
+                    # Prefer "real" vhost paths instead of symlinked ones
+                    # ex: sites-enabled/vh.conf -> sites-available/vh.conf
+
+                    # remove old (most likely) symlinked one
+                    vhs = [v for v in vhs if v.filep != vhost_paths[realpath]]
+                    vhs.append(new_vhost)
+                    vhost_paths[realpath] = realpath
 
         return vhs
 


Test patch:
diff --git a/letsencrypt-apache/letsencrypt_apache/tests/configurator_test.py b/letsencrypt-apache/letsencrypt_apache/tests/configurator_test.py
--- a/letsencrypt-apache/letsencrypt_apache/tests/configurator_test.py
+++ b/letsencrypt-apache/letsencrypt_apache/tests/configurator_test.py
@@ -128,20 +128,10 @@ def test_get_virtual_hosts(self):
         self.assertEqual(found, 6)
 
         # Handle case of non-debian layout get_virtual_hosts
-        orig_conf = self.config.conf
         with mock.patch(
                 "letsencrypt_apache.configurator.ApacheConfigurator.conf"
-        )  as mock_conf:
-            def conf_sideeffect(key):
-                """Handle calls to configurator.conf()
-                :param key: configuration key
-                :return: configuration value
-                """
-                if key == "handle-sites":
-                    return False
-                else:
-                    return orig_conf(key)
-            mock_conf.side_effect = conf_sideeffect
+        ) as mock_conf:
+            mock_conf.return_value = False
             vhs = self.config.get_virtual_hosts()
             self.assertEqual(len(vhs), 6)