Repository: certbot/certbot

Issue:
certbot http -> https rewrite rules messes up GET params
Hello,

An issue I encountered today: the rewrite rules added to apache config by certbot-auto to coerce http requests to https doesn't seem to be bulletproof.

In particular, GET params that are url-encoded, eg ?email=testest%2Byhbugfixer%40gmail.com, seem to end up getting encoded twice so that my application receives 'testest%2Byhbugfixer%40gmail.com' for the param value instead of the intended 'testest+yhbugfixer@gmail.com', if I make the initial request in http://

Notably, this does not happen if in place of the rewrite rules added by certbot-auto, I simply create a new virtual host like so:

 &lt;VirtualHost *:80&gt;
        ServerName mydomain.com
        Redirect / https://mydomain.com/
 &lt;/VirtualHost&gt;



Discussion:
@sagi, thoughts?

@jasonmoofang, can you please share with us your configuration files, before and after certbot messes with them ?

hey @sagi ,

Unfortunately I no longer have the "before" configuration files, but I'm pretty sure this is the snippet inserted by certbot to coerce https:

RewriteEngine on
RewriteCond %{SERVER_NAME} =mydomain.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]

This is the contents of the virtual host file for my domain, if relevant:

  &lt;VirtualHost *:443>
      ServerName mydomain.com
      # !!! Be sure to point DocumentRoot to 'public'!
      DocumentRoot /path/to/rails/public
      RailsEnv production
      &lt;Directory /path/to/rails/public>
         Require all granted
         # This relaxes Apache security settings.
         AllowOverride all
         # MultiViews must be turned off.
         Options -MultiViews
      &lt;/Directory>
      SSLEngine on
      SSLCertificateFile /etc/letsencrypt/live/mydomain.com/cert.pem
      SSLCertificateKeyFile /etc/letsencrypt/live/mydomain.com/privkey.pem
      SSLCertificateChainFile /etc/letsencrypt/live/mydomain.com/chain.pem
   &lt;VirtualHost>

I ran into this problem today.  I think the fix is to add the `NE` (noescape) flag to the rewrite rules found [here](https://github.com/certbot/certbot/blob/777356833221669ce578b53dc81a31100e0511aa/certbot-apache/certbot_apache/constants.py#L138-L146), but I'm not sure if that would create any undesired side effects.

Back then, during my research on this I bumped on this [thread](http://serverfault.com/questions/331899/apache-mod-rewrite-double-encodes-query-string-on-redirect). 

It seems that `NE` introduces undesired side effects, which were not entirely fixed in Apache 2.4.10 (although Apache claims otherwise). 

I've been meaning to test it out; but didn't get to it.

Also, I believe that a change in escaping / encoding of the query string may harm current `certbot` users who already got themselves used to its behavior. 

Double-encoding a query is a well known issue with a resolution to use NE. There are cases where NE isn't the answer. However, for a blind wildcard redirect to move a client from http -> https it is important to not alter any part of the URL beyond the protocol (i.e. path, port, query, etc.).

Failure example:
`http://foo.bar.com/apple?email=me%40domain.com`
incorrectly redirecting to:
`https://foo.bar.com/apple?email=me%2540domain.com`

There is no reliable manner for the receiving code to know if the received query string is single url encoded or erroneously double url encoded. Therefore, there is no legit legacy workaround coded by current certbot users. Anything written by someone out there...isn't reliable.

Therefore, the resolution is to fix the core issue which is the use of QSA. It should instead be NE. This will result in...

Good example:
`http://foo.bar.com/apple?email=me%40domain.com`
correctly redirecting to:
`https://foo.bar.com/apple?email=me%40domain.com`


Resolving patch:
diff --git a/certbot-apache/certbot_apache/configurator.py b/certbot-apache/certbot_apache/configurator.py
--- a/certbot-apache/certbot_apache/configurator.py
+++ b/certbot-apache/certbot_apache/configurator.py
@@ -1315,18 +1315,15 @@ def _enable_redirect(self, ssl_vhost, unused_options):
             #     even with save() and load()
             if not self._is_rewrite_engine_on(general_vh):
                 self.parser.add_dir(general_vh.path, "RewriteEngine", "on")
+
             names = ssl_vhost.get_names()
             for idx, name in enumerate(names):
                 args = ["%{SERVER_NAME}", "={0}".format(name), "[OR]"]
                 if idx == len(names) - 1:
                     args.pop()
                 self.parser.add_dir(general_vh.path, "RewriteCond", args)
-            if self.get_version() >= (2, 3, 9):
-                self.parser.add_dir(general_vh.path, "RewriteRule",
-                                    constants.REWRITE_HTTPS_ARGS_WITH_END)
-            else:
-                self.parser.add_dir(general_vh.path, "RewriteRule",
-                                    constants.REWRITE_HTTPS_ARGS)
+
+            self._set_https_redirection_rewrite_rule(general_vh)
 
             self.save_notes += ("Redirecting host in %s to ssl vhost in %s\n" %
                                 (general_vh.filep, ssl_vhost.filep))
@@ -1336,12 +1333,24 @@ def _enable_redirect(self, ssl_vhost, unused_options):
             logger.info("Redirecting vhost in %s to ssl vhost in %s",
                         general_vh.filep, ssl_vhost.filep)
 
+    def _set_https_redirection_rewrite_rule(self, vhost):
+        if self.get_version() >= (2, 3, 9):
+            self.parser.add_dir(vhost.path, "RewriteRule",
+                    constants.REWRITE_HTTPS_ARGS_WITH_END)
+        else:
+            self.parser.add_dir(vhost.path, "RewriteRule",
+                    constants.REWRITE_HTTPS_ARGS)
+
+
     def _verify_no_certbot_redirect(self, vhost):
         """Checks to see if a redirect was already installed by certbot.
 
         Checks to see if virtualhost already contains a rewrite rule that is
         identical to Certbot's redirection rewrite rule.
 
+        For graceful transition to new rewrite rules for HTTPS redireciton we
+        delete certbot's old rewrite rules and set the new one instead.
+
         :param vhost: vhost to check
         :type vhost: :class:`~certbot_apache.obj.VirtualHost`
 
@@ -1355,19 +1364,29 @@ def _verify_no_certbot_redirect(self, vhost):
         # rewrite_args_dict keys are directive ids and the corresponding value
         # for each is a list of arguments to that directive.
         rewrite_args_dict = defaultdict(list)
-        pat = r'.*(directive\[\d+\]).*'
+        pat = r'(.*directive\[\d+\]).*'
         for match in rewrite_path:
             m = re.match(pat, match)
             if m:
-                dir_id = m.group(1)
-                rewrite_args_dict[dir_id].append(match)
+                dir_path = m.group(1)
+                rewrite_args_dict[dir_path].append(match)
 
         if rewrite_args_dict:
             redirect_args = [constants.REWRITE_HTTPS_ARGS,
                              constants.REWRITE_HTTPS_ARGS_WITH_END]
 
-            for matches in rewrite_args_dict.values():
-                if [self.aug.get(x) for x in matches] in redirect_args:
+            for dir_path, args_paths in rewrite_args_dict.items():
+                arg_vals = [self.aug.get(x) for x in args_paths]
+
+                # Search for past redirection rule, delete it, set the new one
+                if arg_vals in constants.OLD_REWRITE_HTTPS_ARGS:
+                    self.aug.remove(dir_path)
+                    self._set_https_redirection_rewrite_rule(vhost)
+                    self.save()
+                    raise errors.PluginEnhancementAlreadyPresent(
+                        "Certbot has already enabled redirection")
+
+                if arg_vals in redirect_args:
                     raise errors.PluginEnhancementAlreadyPresent(
                         "Certbot has already enabled redirection")
 
diff --git a/certbot-apache/certbot_apache/constants.py b/certbot-apache/certbot_apache/constants.py
--- a/certbot-apache/certbot_apache/constants.py
+++ b/certbot-apache/certbot_apache/constants.py
@@ -136,15 +136,19 @@
 """Path to the Augeas lens directory"""
 
 REWRITE_HTTPS_ARGS = [
-    "^", "https://%{SERVER_NAME}%{REQUEST_URI}", "[L,QSA,R=permanent]"]
+    "^", "https://%{SERVER_NAME}%{REQUEST_URI}", "[L,NE,R=permanent]"]
 """Apache version<2.3.9 rewrite rule arguments used for redirections to
 https vhost"""
 
 REWRITE_HTTPS_ARGS_WITH_END = [
-    "^", "https://%{SERVER_NAME}%{REQUEST_URI}", "[END,QSA,R=permanent]"]
+    "^", "https://%{SERVER_NAME}%{REQUEST_URI}", "[END,NE,R=permanent]"]
 """Apache version >= 2.3.9 rewrite rule arguments used for redirections to
     https vhost"""
 
+OLD_REWRITE_HTTPS_ARGS = [
+    ["^", "https://%{SERVER_NAME}%{REQUEST_URI}", "[L,QSA,R=permanent]"],
+    ["^", "https://%{SERVER_NAME}%{REQUEST_URI}", "[END,QSA,R=permanent]"]]
+
 HSTS_ARGS = ["always", "set", "Strict-Transport-Security",
              "\"max-age=31536000\""]
 """Apache header arguments for HSTS"""


Test patch:
diff --git a/certbot-apache/certbot_apache/tests/configurator_test.py b/certbot-apache/certbot_apache/tests/configurator_test.py
--- a/certbot-apache/certbot_apache/tests/configurator_test.py
+++ b/certbot-apache/certbot_apache/tests/configurator_test.py
@@ -16,6 +16,7 @@
 from certbot.tests import util as certbot_util
 
 from certbot_apache import configurator
+from certbot_apache import constants
 from certbot_apache import parser
 from certbot_apache import obj
 
@@ -1045,6 +1046,36 @@ def test_redirect_with_existing_rewrite(self, mock_exe, _):
 
         self.assertTrue("rewrite_module" in self.config.parser.modules)
 
+    @mock.patch("certbot.util.run_script")
+    @mock.patch("certbot.util.exe_exists")
+    def test_redirect_with_old_https_redirection(self, mock_exe, _):
+        self.config.parser.update_runtime_variables = mock.Mock()
+        mock_exe.return_value = True
+        self.config.get_version = mock.Mock(return_value=(2, 2, 0))
+
+        ssl_vhost = self.config.choose_vhost("certbot.demo")
+
+        # pylint: disable=protected-access
+        http_vhost = self.config._get_http_vhost(ssl_vhost)
+
+        # Create an old (previously suppoorted) https redirectoin rewrite rule
+        self.config.parser.add_dir(
+            http_vhost.path, "RewriteRule",
+            ["^",
+             "https://%{SERVER_NAME}%{REQUEST_URI}",
+             "[L,QSA,R=permanent]"])
+
+        self.config.save()
+
+        try:
+            self.config.enhance("certbot.demo", "redirect")
+        except errors.PluginEnhancementAlreadyPresent:
+            args_paths = self.config.parser.find_dir(
+                "RewriteRule", None, http_vhost.path, False)
+            arg_vals = [self.config.aug.get(x) for x in args_paths]
+            self.assertEqual(arg_vals, constants.REWRITE_HTTPS_ARGS)
+
+
     def test_redirect_with_conflict(self):
         self.config.parser.modules.add("rewrite_module")
         ssl_vh = obj.VirtualHost(
@@ -1132,7 +1163,7 @@ def test_make_vhost_ssl_with_existing_rewrite_rule(self, mock_get_utility):
             http_vhost.path, "RewriteRule",
             ["^",
              "https://%{SERVER_NAME}%{REQUEST_URI}",
-             "[L,QSA,R=permanent]"])
+             "[L,NE,R=permanent]"])
         self.config.save()
 
         ssl_vhost = self.config.make_vhost_ssl(self.vh_truth[0])
@@ -1143,7 +1174,7 @@ def test_make_vhost_ssl_with_existing_rewrite_rule(self, mock_get_utility):
         conf_text = open(ssl_vhost.filep).read()
         commented_rewrite_rule = ("# RewriteRule ^ "
                                   "https://%{SERVER_NAME}%{REQUEST_URI} "
-                                  "[L,QSA,R=permanent]")
+                                  "[L,NE,R=permanent]")
         self.assertTrue(commented_rewrite_rule in conf_text)
         mock_get_utility().add_message.assert_called_once_with(mock.ANY,
 
@@ -1162,7 +1193,7 @@ def test_make_vhost_ssl_with_existing_rewrite_conds(self, mock_get_utility):
                 "RewriteCond", ["%{DOCUMENT_ROOT}/%{REQUEST_FILENAME}", "!-f"])
         self.config.parser.add_dir(
             http_vhost.path, "RewriteRule",
-            ["^(.*)$", "b://u%{REQUEST_URI}", "[P,QSA,L]"])
+            ["^(.*)$", "b://u%{REQUEST_URI}", "[P,NE,L]"])
 
         # Add a chunk that should be commented out.
         self.config.parser.add_dir(http_vhost.path,
@@ -1173,7 +1204,7 @@ def test_make_vhost_ssl_with_existing_rewrite_conds(self, mock_get_utility):
             http_vhost.path, "RewriteRule",
             ["^",
              "https://%{SERVER_NAME}%{REQUEST_URI}",
-             "[L,QSA,R=permanent]"])
+             "[L,NE,R=permanent]"])
 
         self.config.save()
 
@@ -1184,13 +1215,13 @@ def test_make_vhost_ssl_with_existing_rewrite_conds(self, mock_get_utility):
         not_commented_cond1 = ("RewriteCond "
                 "%{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f")
         not_commented_rewrite_rule = ("RewriteRule "
-            "^(.*)$ b://u%{REQUEST_URI} [P,QSA,L]")
+            "^(.*)$ b://u%{REQUEST_URI} [P,NE,L]")
 
         commented_cond1 = "# RewriteCond %{HTTPS} !=on"
         commented_cond2 = "# RewriteCond %{HTTPS} !^$"
         commented_rewrite_rule = ("# RewriteRule ^ "
                                   "https://%{SERVER_NAME}%{REQUEST_URI} "
-                                  "[L,QSA,R=permanent]")
+                                  "[L,NE,R=permanent]")
 
         self.assertTrue(not_commented_cond1 in conf_line_set)
         self.assertTrue(not_commented_rewrite_rule in conf_line_set)