Repository: readthedocs/readthedocs.org

Issue:
Domain resolution too clever
One of the tactics that we use for domain magic can result in some unexpected problems. If the project does not have a domain associated with it yet, our middleware will attempt a DNS lookup. This block will attempt to get a slug from a CNAME:
https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/core/middleware.py#L67-L76

We aren't doing anything to check that the domain is our though. In this case, the domain `virtualenv.pypa.io` resolves as such:

```
% dig +short virtualenv.pypa.io
c.global-ssl.fastly.net.
199.27.79.175
```

The above block will split the CNAME returned, assume the slug is the `c`, and apply the domain to this project if it exists.

Instead, we should probably ensure the CNAME domain is something we control, and this is indeed a slug, not a hostname.



Discussion:
This is also happening in API v2: http://readthedocs.org/api/v2/cname/?host=packaging.python.org

``` json
{"host":"packaging.python.org","slug":"python"}
```

`dig` output for packaging.python.org:

``` console
$ dig +short packaging.python.org
python.map.fastly.net.
prod.python.map.fastlylb.net.
151.101.60.223
```

I've opened https://github.com/rtfd/readthedocs.org/pull/2317 to use the same helper both in middleware and API.

So, I was reading this, I don't fully understand the issue. In what case this bug is triggered? Why some of this domains aren't resolving to *.readthedocs.io (but they are hosted by rtd).

As a solution, we should check that the domain matches `PUBLIC_DOMAIN`, is that correct? But I still have the same doubts as above p:

Also, do we really need the `/cname` endpoint?
So, to trigger this, we have middleware that tries to guess the project based on DNS. I'm not convinced this should be a feature of RTD. The lookup logic is supposed to be:

* See incoming request for domain that isn't our PUBLIC_DOMAIN and isn't already a Domain
* Do a DNS lookup on the domain
* Assume the domain is pointed to `<project_slug>.readthedocs.io` and parse it accordingly
* Bring up the project for `<project_slug>`

When it fails is when the domain we're seeing a request for either isn't actually pointing to us, or when there's some other record, like the DNS request above:

```
% dig packaging.python.org +short
dualstack.python.map.fastly.net.
151.101.52.223
```

Our code assumes the project slug is `dualstack`, ie:
http://readthedocs.org/api/v2/cname/?host=packaging.python.org

So!

* We need to determine what might be removed if we just gutted this feature
* We need to know if there domains that would be affected by removing this feature? We might be logging this currently. Historically I think we made Domain objects that were `machine=True` when we relied on this feature
* I'm not sure what else is using the /cname endpoint, that also needs some discovery

Based on what you find, we are going to be weighing:

* Removing the feature altogether and forcing users to establish domains with us (with cloudflare, I believe this might be an absolute requirement) -- @davidfischer can confirm
* Or do we repair the logic and fix parsing. We are currently asking users to point to `readthedocs.io` I believe in our docs, so this middleware will just be working less and less.

Looks like things could be stacking up to just remove the feature :fire: 
So, this happens if users cname us without creating a domain in the admin panel, right?
> Historically I think we made Domain objects that were machine=True when we relied on this feature

There are ~1k of these.

> Removing the feature altogether and forcing users to establish domains with us (with cloudflare, I believe this might be an absolute requirement) -- @davidfischer can confirm

It is required to generate a certificate but `*.readthedocs.io` goes right to our load balancer and not to Cloudflare. Edit: at least until #4521.

With that said, I'm +1 on removing this feature. If people want to use RTD with a custom domain, I don't think requiring them to put that domain in the dashboard is burdensome.
> So, this happens if users cname us without creating a domain in the admin panel, right?

Correct

> I don't think requiring them to put that domain in the dashboard is burdensome

I'd say it is probably expected even
+1 to removing our old magic logic. We just need to make sure it won't break a bunch of users when we do it. I believe we backported domains that were doing this into `Domain` objects when we first released the explicit `Domain`'s, but we might need to do something similar when we remove the auto-slugify based on DNS, to make sure old domains don't break. 

Resolving patch:
diff --git a/readthedocs/core/middleware.py b/readthedocs/core/middleware.py
--- a/readthedocs/core/middleware.py
+++ b/readthedocs/core/middleware.py
@@ -1,19 +1,14 @@
-# -*- coding: utf-8 -*-
-
-"""Middleware for core app."""
-
 import logging
 
 from django.conf import settings
 from django.contrib.sessions.middleware import SessionMiddleware
-from django.core.cache import cache
 from django.core.exceptions import MultipleObjectsReturned, ObjectDoesNotExist
 from django.http import Http404, HttpResponseBadRequest
 from django.urls.base import set_urlconf
 from django.utils.deprecation import MiddlewareMixin
 from django.utils.translation import ugettext_lazy as _
+from django.shortcuts import render
 
-from readthedocs.core.utils import cname_to_slug
 from readthedocs.projects.models import Domain, Project
 
 
@@ -112,46 +107,21 @@ def process_request(self, request):
                 )
             # Try header first, then DNS
             elif not hasattr(request, 'domain_object'):
-                try:
-                    slug = cache.get(host)
-                    if not slug:
-                        slug = cname_to_slug(host)
-                        cache.set(host, slug, 60 * 60)
-                        # Cache the slug -> host mapping permanently.
-                        log.info(
-                            LOG_TEMPLATE.format(
-                                msg='CNAME cached: {}->{}'.format(slug, host),
-                                **log_kwargs
-                            ),
-                        )
-                    request.slug = slug
-                    request.urlconf = SUBDOMAIN_URLCONF
-                    log.warning(
-                        LOG_TEMPLATE.format(
-                            msg='CNAME detected: %s' % request.slug,
-                            **log_kwargs
-                        ),
-                    )
-                except:  # noqa
-                    # Some crazy person is CNAMEing to us. 404.
-                    log.warning(
-                        LOG_TEMPLATE.format(msg='CNAME 404', **log_kwargs),
-                    )
-                    raise Http404(_('Invalid hostname'))
+                # Some person is CNAMEing to us without configuring a domain - 404.
+                log.warning(LOG_TEMPLATE.format(msg='CNAME 404', **log_kwargs))
+                return render(request, 'core/dns-404.html', context={'host': host}, status=404)
         # Google was finding crazy www.blah.readthedocs.org domains.
         # Block these explicitly after trying CNAME logic.
         if len(domain_parts) > 3 and not settings.DEBUG:
             # Stop www.fooo.readthedocs.org
             if domain_parts[0] == 'www':
-                log.debug(
-                    LOG_TEMPLATE.format(msg='404ing long domain', **log_kwargs),
-                )
+                log.debug(LOG_TEMPLATE.format(
+                    msg='404ing long domain', **log_kwargs
+                ))
                 return HttpResponseBadRequest(_('Invalid hostname'))
-            log.debug(
-                LOG_TEMPLATE
-                .format(msg='Allowing long domain name', **log_kwargs),
-            )
-            # raise Http404(_('Invalid hostname'))
+            log.debug(LOG_TEMPLATE.format(
+                msg='Allowing long domain name', **log_kwargs
+            ))
         # Normal request.
         return None
 
diff --git a/readthedocs/core/utils/__init__.py b/readthedocs/core/utils/__init__.py
--- a/readthedocs/core/utils/__init__.py
+++ b/readthedocs/core/utils/__init__.py
@@ -59,15 +59,6 @@ def broadcast(type, task, args, kwargs=None, callback=None):  # pylint: disable=
     return task_promise
 
 
-def cname_to_slug(host):
-    # TODO: remove
-    from dns import resolver
-    answer = [ans for ans in resolver.query(host, 'CNAME')][0]
-    domain = answer.target.to_unicode()
-    slug = domain.split('.')[0]
-    return slug
-
-
 def prepare_build(
         project,
         version=None,


Test patch:
diff --git a/readthedocs/rtd_tests/tests/test_middleware.py b/readthedocs/rtd_tests/tests/test_middleware.py
--- a/readthedocs/rtd_tests/tests/test_middleware.py
+++ b/readthedocs/rtd_tests/tests/test_middleware.py
@@ -1,15 +1,11 @@
-# -*- coding: utf-8 -*-
-
 from corsheaders.middleware import CorsMiddleware
 from django.conf import settings
-from django.core.cache import cache
 from django.http import Http404
 from django.test import TestCase
 from django.test.client import RequestFactory
 from django.test.utils import override_settings
 from django.urls.base import get_urlconf, set_urlconf
 from django_dynamic_fixture import get
-from mock import patch
 
 from readthedocs.core.middleware import SubdomainMiddleware
 from readthedocs.projects.models import Domain, Project, ProjectRelationship
@@ -26,12 +22,18 @@ def setUp(self):
         self.middleware = SubdomainMiddleware()
         self.url = '/'
         self.owner = create_user(username='owner', password='test')
-        self.pip = get(Project, slug='pip', users=[self.owner], privacy_level='public')
+        self.pip = get(
+            Project,
+            slug='pip',
+            users=[self.owner],
+            privacy_level='public'
+        )
 
     def test_failey_cname(self):
+        self.assertFalse(Domain.objects.filter(domain='my.host.com').exists())
         request = self.factory.get(self.url, HTTP_HOST='my.host.com')
-        with self.assertRaises(Http404):
-            self.middleware.process_request(request)
+        r = self.middleware.process_request(request)
+        self.assertEqual(r.status_code, 404)
         self.assertEqual(request.cname, True)
 
     @override_settings(PRODUCTION_DOMAIN='readthedocs.org')
@@ -74,7 +76,9 @@ def test_restore_urlconf_after_request(self):
 
     @override_settings(PRODUCTION_DOMAIN='prod.readthedocs.org')
     def test_subdomain_different_length(self):
-        request = self.factory.get(self.url, HTTP_HOST='pip.prod.readthedocs.org')
+        request = self.factory.get(
+            self.url, HTTP_HOST='pip.prod.readthedocs.org'
+        )
         self.middleware.process_request(request)
         self.assertEqual(request.urlconf, self.urlconf_subdomain)
         self.assertEqual(request.subdomain, True)
@@ -92,19 +96,13 @@ def test_domain_object(self):
     def test_domain_object_missing(self):
         self.domain = get(Domain, domain='docs.foobar2.com', project=self.pip)
         request = self.factory.get(self.url, HTTP_HOST='docs.foobar.com')
-        with self.assertRaises(Http404):
-            self.middleware.process_request(request)
-
-    def test_proper_cname(self):
-        cache.get = lambda x: 'my_slug'
-        request = self.factory.get(self.url, HTTP_HOST='my.valid.homename')
-        self.middleware.process_request(request)
-        self.assertEqual(request.urlconf, self.urlconf_subdomain)
-        self.assertEqual(request.cname, True)
-        self.assertEqual(request.slug, 'my_slug')
+        r = self.middleware.process_request(request)
+        self.assertEqual(r.status_code, 404)
 
     def test_request_header(self):
-        request = self.factory.get(self.url, HTTP_HOST='some.random.com', HTTP_X_RTD_SLUG='pip')
+        request = self.factory.get(
+            self.url, HTTP_HOST='some.random.com', HTTP_X_RTD_SLUG='pip'
+        )
         self.middleware.process_request(request)
         self.assertEqual(request.urlconf, self.urlconf_subdomain)
         self.assertEqual(request.cname, True)
@@ -113,7 +111,7 @@ def test_request_header(self):
 
     @override_settings(PRODUCTION_DOMAIN='readthedocs.org')
     def test_proper_cname_uppercase(self):
-        cache.get = lambda x: x.split('.')[0]
+        get(Domain, project=self.pip, domain='pip.random.com')
         request = self.factory.get(self.url, HTTP_HOST='PIP.RANDOM.COM')
         self.middleware.process_request(request)
         self.assertEqual(request.urlconf, self.urlconf_subdomain)
@@ -121,20 +119,35 @@ def test_proper_cname_uppercase(self):
         self.assertEqual(request.slug, 'pip')
 
     def test_request_header_uppercase(self):
-        request = self.factory.get(self.url, HTTP_HOST='some.random.com', HTTP_X_RTD_SLUG='PIP')
+        request = self.factory.get(
+            self.url, HTTP_HOST='some.random.com', HTTP_X_RTD_SLUG='PIP'
+        )
         self.middleware.process_request(request)
         self.assertEqual(request.urlconf, self.urlconf_subdomain)
         self.assertEqual(request.cname, True)
         self.assertEqual(request.rtdheader, True)
         self.assertEqual(request.slug, 'pip')
 
-    @override_settings(USE_SUBDOMAIN=True)
-    # no need to do a real dns query so patch cname_to_slug
-    @patch('readthedocs.core.middleware.cname_to_slug', new=lambda x: 'doesnt')
-    def test_use_subdomain_on(self):
-        request = self.factory.get(self.url, HTTP_HOST='doesnt.really.matter')
-        ret_val = self.middleware.process_request(request)
-        self.assertIsNone(ret_val, None)
+    def test_use_subdomain(self):
+        domain = 'doesnt.exists.org'
+        get(Domain, project=self.pip, domain=domain)
+        request = self.factory.get(self.url, HTTP_HOST=domain)
+        res = self.middleware.process_request(request)
+        self.assertIsNone(res)
+        self.assertEqual(request.slug, 'pip')
+        self.assertTrue(request.domain_object)
+
+    def test_long_bad_subdomain(self):
+        domain = 'www.pip.readthedocs.org'
+        request = self.factory.get(self.url, HTTP_HOST=domain)
+        res = self.middleware.process_request(request)
+        self.assertEqual(res.status_code, 400)
+
+    def test_long_subdomain(self):
+        domain = 'some.long.readthedocs.org'
+        request = self.factory.get(self.url, HTTP_HOST=domain)
+        res = self.middleware.process_request(request)
+        self.assertIsNone(res)
 
 
 class TestCORSMiddleware(TestCase):