Repository: readthedocs/readthedocs.org

Issue:
Serve custom 404 pages from Django
Currently, we are serving the `404.html` from NGINX returning a `Response` that includes the `X-Accel-Redirect` header:

https://github.com/rtfd/readthedocs.org/blob/537f7b8d2e8d81f8243d1d4a562968e472bc6337/readthedocs/core/views/__init__.py#L208-L210

We found that even marking the `response.status_code = 404`, the response is 200.

```
$ curl -IL https://docs.readthedocs.io/notfound
HTTP/1.1 200 OK
Content-Length: 14167
Content-Type: text/html
Last-Modified: Mon, 22 Apr 2019 15:30:57 GMT
Accept-Ranges: bytes
ETag: "5cbdde31-3757"
Vary: Accept-Encoding
Server: nginx
X-Served: Nginx-Sendfile
X-Deity: web01
Strict-Transport-Security: max-age=1800; includeSubDomains
Date: Mon, 29 Apr 2019 07:59:21 GMT
```

To fix this, we have to serve this file directly from Django instead of using this header.


Discussion:
Working on it! 
I looked into it. I'm a bit confused. can you tell me why this test passes:
`self.assertEqual(response.status_code, 404)`
https://github.com/rtfd/readthedocs.org/blob/537f7b8d2e8d81f8243d1d4a562968e472bc6337/readthedocs/rtd_tests/tests/test_doc_serving.py#L206-L224

and do we need to use the `serve()` or `render()` instead of `_serve_file()` method from django to serve this?
The test passes because Django is returning a 404 properly. Although, in production, that response is passed to NGINX who looks for the `X-Accel-Redirect` header and serve that file directly from disk. At this point, because NGINX found that file and was able to serve it properly, it adds the 200 in the response.

As we don't use NGINX at all for our tests, it succees.

> and do we need to use the `serve()` or `render()` instead of `_serve_file()` method from django to serve this?

I suppose we need to use `serve` since it's a regular file, not a template.

Resolving patch:
diff --git a/readthedocs/core/views/__init__.py b/readthedocs/core/views/__init__.py
--- a/readthedocs/core/views/__init__.py
+++ b/readthedocs/core/views/__init__.py
@@ -14,13 +14,12 @@
 from django.http import HttpResponseRedirect, Http404, JsonResponse
 from django.shortcuts import render, get_object_or_404, redirect
 from django.views.generic import TemplateView
-
+from django.views.static import serve as static_serve
 
 from readthedocs.builds.models import Version
 from readthedocs.core.utils.general import wipe_version_via_slugs
 from readthedocs.core.resolver import resolve_path
 from readthedocs.core.symlink import PrivateSymlink, PublicSymlink
-from readthedocs.core.views.serve import _serve_file
 from readthedocs.projects.constants import PRIVATE
 from readthedocs.projects.models import HTMLFile, Project
 from readthedocs.redirects.utils import (
@@ -205,7 +204,7 @@ def resolve_404_path(project, version_slug=None, language=None):
                     project.slug,
                     slug,
                 )
-                r = _serve_file(request, filename, basepath)
+                r = static_serve(request, filename, basepath)
                 r.status_code = 404
                 return r
 


Test patch:
diff --git a/readthedocs/rtd_tests/tests/test_doc_serving.py b/readthedocs/rtd_tests/tests/test_doc_serving.py
--- a/readthedocs/rtd_tests/tests/test_doc_serving.py
+++ b/readthedocs/rtd_tests/tests/test_doc_serving.py
@@ -201,14 +201,11 @@ def test_custom_robots_txt(self, os_mock, open_mock):
         ROOT_URLCONF=settings.SUBDOMAIN_URLCONF,
     )
 
-    @patch('readthedocs.core.views.serve.os')
+    @patch('readthedocs.core.views.static_serve')
     @patch('readthedocs.core.views.os')
-    def test_custom_404_page(self, os_view_mock, os_serve_mock):
+    def test_custom_404_page(self, os_view_mock, static_serve_mock):
         os_view_mock.path.exists.return_value = True
 
-        os_serve_mock.path.join.side_effect = os.path.join
-        os_serve_mock.path.exists.return_value = True
-
         self.public.versions.update(active=True, built=True)
 
         factory = RequestFactory()
@@ -221,7 +218,6 @@ def test_custom_404_page(self, os_view_mock, os_serve_mock):
         middleware.process_request(request)
         response = server_error_404_subdomain(request)
         self.assertEqual(response.status_code, 404)
-        self.assertTrue(response['X-Accel-Redirect'].endswith('/public/en/latest/404.html'))
 
     @override_settings(
         USE_SUBDOMAIN=True,