Repository: certbot/certbot

Issue:
apache parser searching sites-available, should be sites-enabled
for obvious reasons..



Discussion:
Right now, the parser searches both... sites-enabled for what configuration directives are actually enabled and sites-available for all of the possible sites that you may have configured, for VirtualHost selection.

My idea was that Apache can automatically enable a site that may have been created, but not yet deployed yet.  If Apache is configured appropriately for Debian, sites-enabled should be a subset of sites-available.

The Apache VirtualHost objects have an attribute for if the site is currently enabled.

I do think that it makes sense to completely "generalize" the configuration by only searching sites-enabled (it would make porting to other layouts easier).

I would suggest that we don't make a distinction between sites-enabled and any other Included directory.  We should simply look through all of the current configuration files and look for VirtualHost blocks.

ah, so this is relying on sites-enabled being symlinks to sites-available then?

@jdkasten so the only thing that would break if we remove sites-available from the search path is enablement of vhosts that the admin has set up but not actually deployed?  In general that sounds slightly dangerous anyway, so if this is a simple fix we should just remove `sites-available` from whatever calls construct the search path.

@BillKeenan interested in making a PR?

I am. on it.

does anyone have some more fullsome docs on getting this running with vagrant.. currently getting this

```
Error: unauthorized :: The client lacks sufficient authorization :: Error creating new authz :: Name is blacklisted

```

after issuing this; sudo ./venv/bin/letsencrypt -d www.example.com auth

`www.example.com` is blacklisted, use real domain

this prevents me from using letsencrypt, i have no files in sites-available and the letsencrypt client tells me that no config was found and closes.

you can use the manual authorization mode until this is resolved., i did, worked well/easy.

We should definitely fix this!  @BillKeenan are you still interested in writing this PR? If not, we should assign someone the ticket...

right! Yes! I have it mostly in the can, I'll get on it tonight.

@BillKeenan nudge :)

hmm, i just ran it, and it's pulling a site from my sites-enabled without issue...

@BillKeenan do you mean you can no longer reproduce this bug, or that you have a branch that fixes it?  I can definitely still reproduce...

Can you provide steps to reproduce?

Bill Keenan
VP Technology
416.583.3772
bill@authenticweb.com

On 4 Jan 2016, at 10:37, Peter Eckersley wrote:

> @BillKeenan do you mean you can no longer reproduce this bug, or that 
> you have a branch that fixes it?  I can definitely still reproduce...
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/letsencrypt/letsencrypt/issues/1043#issuecomment-168708754

Sure!  Though I think @joohoi might have started working on this, so it may not be worth spending more time on at the moment. 

```
pde@damoclid:~/letsencrypt/letsencrypt$ ls /etc/apache2/sites-*
/etc/apache2/sites-available:

/etc/apache2/sites-enabled:
001-dee.conf  001-tweedle.conf
pde@damoclid:~/letsencrypt/letsencrypt$ cat /etc/apache2/sites-enabled/*
<VirtualHost *:80>
    ServerAdmin webmaster@localhost
        ServerName dee.isnot.org

    DocumentRoot /var/www/tweedle
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /var/www/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>

    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Order allow,deny
        Allow from all
    </Directory>

    <If "%{REQUEST_URI} =~ /thing/ || %{QUERY_STRING} == 'edit=1'">
        RewriteEngine on
        AuthType basic
        AuthBasicProvider file
        AuthUserFile "/etc/apache2/passwords"
        Require valid-user
    </If>


    ErrorLog ${APACHE_LOG_DIR}/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
<VirtualHost *:80>
    ServerAdmin webmaster@localhost
        ServerName tweedle.isnot.org

    DocumentRoot /var/www/tweedle
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /var/www/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>

    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Order allow,deny
        Allow from all
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
pde@damoclid:~/letsencrypt/letsencrypt$ sudo venv/bin/letsencrypt --apache -t

-------------------------------------------------------------------------------
No names were found in your configuration files.
You should specify ServerNames in your config files in order to allow for
accurate installation of your certificate.
If you do use the default vhost, you may specify the name manually. Would you
like to continue?
-------------------------------------------------------------------------------
(Y)es/(N)o:
```

Flagging the ticket as joohoi's for the time being.


Resolving patch:
diff --git a/letsencrypt-apache/letsencrypt_apache/parser.py b/letsencrypt-apache/letsencrypt_apache/parser.py
--- a/letsencrypt-apache/letsencrypt_apache/parser.py
+++ b/letsencrypt-apache/letsencrypt_apache/parser.py
@@ -35,6 +35,7 @@ def __init__(self, aug, root, vhostroot, version=(2, 4)):
         # https://httpd.apache.org/docs/2.4/mod/core.html#define
         # https://httpd.apache.org/docs/2.4/mod/core.html#ifdefine
         # This only handles invocation parameters and Define directives!
+        self.parser_paths = {}
         self.variables = {}
         if version >= (2, 4):
             self.update_runtime_variables()
@@ -471,16 +472,63 @@ def _parse_file(self, filepath):
         :param str filepath: Apache config file path
 
         """
+        use_new, remove_old = self._check_path_actions(filepath)
         # Test if augeas included file for Httpd.lens
         # Note: This works for augeas globs, ie. *.conf
-        inc_test = self.aug.match(
-            "/augeas/load/Httpd/incl [. ='%s']" % filepath)
-        if not inc_test:
-            # Load up files
-            # This doesn't seem to work on TravisCI
-            # self.aug.add_transform("Httpd.lns", [filepath])
-            self._add_httpd_transform(filepath)
-            self.aug.load()
+        if use_new:
+            inc_test = self.aug.match(
+                "/augeas/load/Httpd/incl [. ='%s']" % filepath)
+            if not inc_test:
+                # Load up files
+                # This doesn't seem to work on TravisCI
+                # self.aug.add_transform("Httpd.lns", [filepath])
+                if remove_old:
+                    self._remove_httpd_transform(filepath)
+                self._add_httpd_transform(filepath)
+                self.aug.load()
+
+    def _check_path_actions(self, filepath):
+        """Determine actions to take with a new augeas path
+
+        This helper function will return a tuple that defines
+        if we should try to append the new filepath to augeas
+        parser paths, and / or remove the old one with more
+        narrow matching.
+
+        :param str filepath: filepath to check the actions for
+
+        """
+
+        try:
+            new_file_match = os.path.basename(filepath)
+            existing_matches = self.parser_paths[os.path.dirname(filepath)]
+            if "*" in existing_matches:
+                use_new = False
+            else:
+                use_new = True
+            if new_file_match == "*":
+                remove_old = True
+            else:
+                remove_old = False
+        except KeyError:
+            use_new = True
+            remove_old = False
+        return use_new, remove_old
+
+    def _remove_httpd_transform(self, filepath):
+        """Remove path from Augeas transform
+
+        :param str filepath: filepath to remove
+        """
+
+        remove_basenames = self.parser_paths[os.path.dirname(filepath)]
+        remove_dirname = os.path.dirname(filepath)
+        for name in remove_basenames:
+            remove_path = remove_dirname + "/" + name
+            remove_inc = self.aug.match(
+                "/augeas/load/Httpd/incl [. ='%s']" % remove_path)
+            self.aug.remove(remove_inc[0])
+        self.parser_paths.pop(remove_dirname)
 
     def _add_httpd_transform(self, incl):
         """Add a transform to Augeas.
@@ -502,6 +550,13 @@ def _add_httpd_transform(self, incl):
             # Augeas uses base 1 indexing... insert at beginning...
             self.aug.set("/augeas/load/Httpd/lens", "Httpd.lns")
             self.aug.set("/augeas/load/Httpd/incl", incl)
+        # Add included path to paths dictionary
+        try:
+            self.parser_paths[os.path.dirname(incl)].append(
+                os.path.basename(incl))
+        except KeyError:
+            self.parser_paths[os.path.dirname(incl)] = [
+                os.path.basename(incl)]
 
     def standardize_excl(self):
         """Standardize the excl arguments for the Httpd lens in Augeas.


Test patch:
diff --git a/letsencrypt-apache/letsencrypt_apache/tests/parser_test.py b/letsencrypt-apache/letsencrypt_apache/tests/parser_test.py
--- a/letsencrypt-apache/letsencrypt_apache/tests/parser_test.py
+++ b/letsencrypt-apache/letsencrypt_apache/tests/parser_test.py
@@ -36,7 +36,7 @@ def test_parse_file(self):
 
         """
         file_path = os.path.join(
-            self.config_path, "sites-available", "letsencrypt.conf")
+            self.config_path, "not-parsed-by-default", "letsencrypt.conf")
 
         self.parser._parse_file(file_path)  # pylint: disable=protected-access