Repository: readthedocs/readthedocs.org

Issue:
Custom robots.txt support?
We've talked about blowing away the protected designation, so not sure if it makes sense to put special case on the protected privacy level, but maybe a separate option for docs that shouldn't be crawled?


Discussion:
@agjohnson any momentum on this particular item? What is the current recommendation to NOINDEX/NOFOLLOW a site?
At very least, we could kill our global robots.txt redirect in nginx and allow projects to contribute their own robots.txt via a static page in Sphinx
@agjohnson what's the status of this issue?

I'm not sure to clearly understand what's the action needed here.

1. if it's around Protected Privacy level, I think we can close it as won't fix since we are removing the privacy levels from Community site.
1. if it's about giving our users a way to upload by themselves a `robots.txt` I think the solution that I proposed at https://github.com/rtfd/readthedocs.org/issues/2430#issuecomment-418471125 should work (there is an example of a repository in that conversation also) and we can close this issue.

If none of those are what you have in mind, please elaborate a little more what you are considering here.

@humitos the solution provided in [#2430 (comment)](https://github.com/rtfd/readthedocs.org/issues/2430#issuecomment-418471125) is not optimal:

> 1. Your site can have only one robots.txt file.
> 2. The robots.txt file must be located at the root of the website host that it applies to. For instance, to control crawling on all URLs below http://www.example.com/, the robots.txt file must be located at http://www.example.com/robots.txt. It cannot be placed in a subdirectory ( for example, at http://example.com/pages/robots.txt). If you're unsure about how to access your website root, or need permissions to do so, contact your web hosting service provider. If you can't access your website root, use an alternative blocking method such as meta tags.
>
> [Google support](https://support.google.com/webmasters/answer/6062596?hl=en)

I think the only viable option is using the "meta tags" method [[1]](http://www.robotstxt.org/meta.html)[[2]](https://support.google.com/webmasters/answer/93710). I am working on a workaround for [Astropy's docs](https://astropy.readthedocs.io/en/stable/) (refer to issue [#7794](https://github.com/astropy/astropy/issues/7794) and pull request [#7874](https://github.com/astropy/astropy/pull/787)).

I'll be done by the end of the day and will let you know. If it's a good workaround. I'd be happy to document the process.
@dasdachs I see. You are right.

> I'll be done by the end of the day and will let you know. If it's a good workaround. I'd be happy to document the process.

If the workaround by using meta tags is a good one, maybe it's a good solution to be implemented by a sphinx extension. It's still a hack, but at least "an automatic one" :grimacing: 

After reading the docs you linked, I don't see a solution coming from Sphinx or without a hack, so I think we should implement this from Read the Docs itself by adding a `robotstxt_file:` option in our YAML (or similar) and copy it at the root of the subdomain. Not sure if that's possible, though.
> I think we should implement this from Read the Docs itself by adding a `robotstxt_file:` option in our YAML

This is not trivial.

With that file, we will need to do:

1. append our [own set of rules](https://github.com/rtfd/readthedocs.org/blob/b07c3a383cc89eb0c92f6070fe888c988da48089/media/robots.txt) to the custom `robots.txt`
1. sync the result to all our web servers
   * since this file will be _outside_ Sphinx output, we need adapt that code
2. modify the nginx rule to try serving first the custom `robots.txt` from the project/version and as a fallback serve ours

This raise another problem: we have one subdomain with multiples versions but only _one_ root place to serve the `robots.txt` file. Which one should we serve?

Being a "global setting" makes me doubt if it isn't better to add a text box in the admin where the user can paste the contents of that file or think something easier like that.
> I think we should implement this from Read the Docs itself by adding a robotstxt_file: option in our YAML

I doubt this will be on the yaml, as this is a per-project configuration rather than per-version
The hack I found could be quite simple ([this](https://github.com/astropy/astropy/pull/7874#issuecomment-429138942)): add meta tags to files you don't want indexed. 
But because of the global `robots.txt`, it would have no affect (refering to this answer from [Google](https://support.google.com/webmasters/answer/93710)). Some solution using YAML or a text box seems like the way to go.
Unfortunately, the idea of adding meta tags isn't really an ideal solution, because we can't add it to all the old versions we host. In the case of astropy for example, we host a lot of old versions based on GitHub tags, e.g.:

http://docs.astropy.org/en/v1.0/

We can't change all the tags in our GitHub repo for all the old versions, so any solution that involves changes to the repository are a no-go. The only real solution would be to be able to customize robots.txt from the RTD settings interface.

Resolving patch:
diff --git a/readthedocs/core/urls/subdomain.py b/readthedocs/core/urls/subdomain.py
--- a/readthedocs/core/urls/subdomain.py
+++ b/readthedocs/core/urls/subdomain.py
@@ -1,3 +1,5 @@
+# -*- coding: utf-8 -*-
+
 """URL configurations for subdomains."""
 from __future__ import absolute_import
 
@@ -10,7 +12,7 @@
 
 from readthedocs.core.views.serve import (
     redirect_page_with_filename,
-    redirect_project_slug, serve_docs
+    redirect_project_slug, serve_docs, robots_txt,
 )
 from readthedocs.core.views import (
     server_error_500,
@@ -22,6 +24,8 @@
 handler404 = server_error_404
 
 subdomain_urls = [
+    url(r'robots.txt$', robots_txt, name='robots_txt'),
+
     url(r'^(?:|projects/(?P<subproject_slug>{project_slug})/)'
         r'page/(?P<filename>.*)$'.format(**pattern_opts),
         redirect_page_with_filename,
diff --git a/readthedocs/core/views/serve.py b/readthedocs/core/views/serve.py
--- a/readthedocs/core/views/serve.py
+++ b/readthedocs/core/views/serve.py
@@ -223,3 +223,49 @@ def _serve_symlink_docs(request, project, privacy_level, filename=''):
 
     raise Http404(
         'File not found. Tried these files: %s' % ','.join(files_tried))
+
+
+@map_project_slug
+def robots_txt(request, project):
+    """
+    Serve custom user's defined ``/robots.txt``.
+
+    If the user added a ``robots.txt`` in the "default version" of the project,
+    we serve it directly.
+    """
+    # Use the ``robots.txt`` file from the default version configured
+    version_slug = project.get_default_version()
+    version = project.versions.get(slug=version_slug)
+
+    no_serve_robots_txt = any([
+        # If project is private or,
+        project.privacy_level == constants.PRIVATE,
+        # default version is private or,
+        version.privacy_level == constants.PRIVATE,
+        # default version is not active or,
+        not version.active,
+        # default version is not built
+        not version.built,
+    ])
+    if no_serve_robots_txt:
+        # ... we do return a 404
+        raise Http404()
+
+    filename = resolve_path(
+        project,
+        version_slug=version_slug,
+        filename='robots.txt',
+        subdomain=True,  # subdomain will make it a "full" path without a URL prefix
+    )
+
+    # This breaks path joining, by ignoring the root when given an "absolute" path
+    if filename[0] == '/':
+        filename = filename[1:]
+
+    basepath = PublicSymlink(project).project_root
+    fullpath = os.path.join(basepath, filename)
+
+    if os.path.exists(fullpath):
+        return HttpResponse(open(fullpath).read(), content_type='text/plain')
+
+    return HttpResponse('User-agent: *\nAllow: /\n', content_type='text/plain')


Test patch:
diff --git a/readthedocs/rtd_tests/tests/test_doc_serving.py b/readthedocs/rtd_tests/tests/test_doc_serving.py
--- a/readthedocs/rtd_tests/tests/test_doc_serving.py
+++ b/readthedocs/rtd_tests/tests/test_doc_serving.py
@@ -2,13 +2,17 @@
 
 from __future__ import absolute_import, unicode_literals, division, print_function
 import mock
+from mock import patch, mock_open
 import django_dynamic_fixture as fixture
+import pytest
+import six
 
 from django.contrib.auth.models import User
 from django.test import TestCase
 from django.test.utils import override_settings
 from django.http import Http404
 from django.conf import settings
+from django.urls import reverse
 
 from readthedocs.rtd_tests.base import RequestFactoryTestMixin
 from readthedocs.projects import constants
@@ -77,6 +81,28 @@ def test_private_files_not_found(self):
         self.assertTrue('private_web_root' in str(exc.exception))
         self.assertTrue('public_web_root' not in str(exc.exception))
 
+    @override_settings(
+        PYTHON_MEDIA=False,
+        USE_SUBDOMAIN=True,
+        PUBLIC_DOMAIN='readthedocs.io',
+        ROOT_URLCONF=settings.SUBDOMAIN_URLCONF,
+    )
+    def test_robots_txt(self):
+        self.public.versions.update(active=True, built=True)
+        response = self.client.get(
+            reverse('robots_txt'),
+            HTTP_HOST='private.readthedocs.io',
+        )
+        self.assertEqual(response.status_code, 404)
+
+        self.client.force_login(self.eric)
+        response = self.client.get(
+            reverse('robots_txt'),
+            HTTP_HOST='private.readthedocs.io',
+        )
+        # Private projects/versions always return 404 for robots.txt
+        self.assertEqual(response.status_code, 404)
+
 
 @override_settings(SERVE_DOCS=[constants.PRIVATE, constants.PUBLIC])
 class TestPublicDocs(BaseDocServing):
@@ -110,3 +136,41 @@ def test_both_files_not_found(self):
             _serve_symlink_docs(request, project=self.private, filename='/en/latest/usage.html', privacy_level='public')
         self.assertTrue('private_web_root' not in str(exc.exception))
         self.assertTrue('public_web_root' in str(exc.exception))
+
+    @override_settings(
+        PYTHON_MEDIA=False,
+        USE_SUBDOMAIN=True,
+        PUBLIC_DOMAIN='readthedocs.io',
+        ROOT_URLCONF=settings.SUBDOMAIN_URLCONF,
+    )
+    def test_default_robots_txt(self):
+        self.public.versions.update(active=True, built=True)
+        response = self.client.get(
+            reverse('robots_txt'),
+            HTTP_HOST='public.readthedocs.io',
+        )
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.content, b'User-agent: *\nAllow: /\n')
+
+    @override_settings(
+        PYTHON_MEDIA=False,
+        USE_SUBDOMAIN=True,
+        PUBLIC_DOMAIN='readthedocs.io',
+        ROOT_URLCONF=settings.SUBDOMAIN_URLCONF,
+    )
+    @patch(
+        'builtins.open',
+        new_callable=mock_open,
+        read_data='My own robots.txt',
+    )
+    @patch('readthedocs.core.views.serve.os')
+    @pytest.mark.skipif(six.PY2, reason='In Python2 the mock is __builtins__.open')
+    def test_custom_robots_txt(self, os_mock, open_mock):
+        os_mock.path.exists.return_value = True
+        self.public.versions.update(active=True, built=True)
+        response = self.client.get(
+            reverse('robots_txt'),
+            HTTP_HOST='public.readthedocs.io',
+        )
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.content, b'My own robots.txt')