Repository: readthedocs/readthedocs.org

Issue:
Remove old symlink when Domain is changed
I started getting random 404 at https://docs.pytest.org/en/latest/ and we found they have confifured `doc.pytest.org` as a their domain so, the new web server was failing on serving the files.

At some point, they should have configured `docs.pytest.org` under RTD and then they changed it to just `doc`, otherwise RTD should know nothing about the `docs` domain in the old web servers.

To fix this, once the Domain is changed we need to perform a cleanup of the old symlinks in the web servers. This can be done by a task that catches all the various symlink leftovers we have, and sync’s them.

* https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/views/private.py#L678-L701
* https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/forms.py#L546


> As a reference, they are mixing `doc` and `docs` in their repo: https://github.com/pytest-dev/pytest/blob/0a15edd5732578f6cc40e28770f0fc87dfa5ced9/doc/en/announce/release-3.2.0.rst


Discussion:
We should do two things:

* Change the logic of the Domain Form or View to trigger a delete symlink task
* Have a background task that runs every hour that checks for invalid symlinks, then deletes them if found

The change should be immediate when users act, but it's nice to have something in the background doing cleanup. It also catches a number of cases the Views & Forms don't catch (eg. if we edit it in the Admin, CLI, ORM `update()` etc.). 
It looks like our symlink domain logic already accounts for deletions, we just aren't calling it: https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/tasks.py#L775-L801
I'm not really sure how the `Domain` works under RTD, but I see there is a `count` attribute (Number of times this domain has been hit) in the model that won't have more sense if we _modify/edit_ the domain object.

Taking a look at the code, it doesn't seem to be tought to allow the _edit_ operation. It looks like more _delete_ and _create_ a new one. In this case, the `delete` method will remove the old symlinks properly since it's calling the proper task.

https://github.com/rtfd/readthedocs.org/blob/1af444173481df9a7b16e015fc6b12abe8155e7e/readthedocs/projects/models.py#L1014-L1018

Regarding the background task, what's the proper way to do this? How do we search for orphan domains?

1. list all the symlinks under `(Public/Private)Symlink.CNAME_ROOT`
1. use each of them to query the database and check the Domain object exists
1. if it doesn't exist, remove the symlink

Sounds good?
To modify the Domain Form or view, we will need to modify the `symlink_domain` task to accept a `domain_name` instead of a `domain_pk` since the domain name that we want to remove doesn't exist anymore in the database but in the filesystem. So, at the moment we don't have a way to remove a "domain name" that it's not a "domain object" in the database.

I see two possible solutions:

1. do not allow _edit_ at all and force the user to delete and create a new one
1. broadcast the periodic task immediately when the domain is modified (kind of overkill)

I think that 1) makes more sense after reading the code.

Resolving patch:
diff --git a/readthedocs/projects/tasks.py b/readthedocs/projects/tasks.py
--- a/readthedocs/projects/tasks.py
+++ b/readthedocs/projects/tasks.py
@@ -828,6 +828,16 @@ def remove_orphan_symlinks():
                 os.unlink(orphan_domain_path)
 
 
+@app.task(queue='web')
+def broadcast_remove_orphan_symlinks():
+    """
+    Broadcast the task ``remove_orphan_symlinks`` to all our web servers.
+
+    This task is executed by CELERY BEAT.
+    """
+    broadcast(type='web', task=remove_orphan_symlinks, args=[])
+
+
 @app.task(queue='web')
 def symlink_subproject(project_pk):
     project = Project.objects.get(pk=project_pk)
diff --git a/readthedocs/settings/base.py b/readthedocs/settings/base.py
--- a/readthedocs/settings/base.py
+++ b/readthedocs/settings/base.py
@@ -247,7 +247,7 @@ def USE_PROMOS(self):  # noqa
     CELERYBEAT_SCHEDULE = {
         # Ran every hour on minute 30
         'hourly-remove-orphan-symlinks': {
-            'task': 'readthedocs.projects.tasks.remove_orphan_symlinks',
+            'task': 'readthedocs.projects.tasks.broadcast_remove_orphan_symlinks',
             'schedule': crontab(minute=30),
             'options': {'queue': 'web'},
         },


Test patch:
diff --git a/readthedocs/rtd_tests/tests/test_project_symlinks.py b/readthedocs/rtd_tests/tests/test_project_symlinks.py
--- a/readthedocs/rtd_tests/tests/test_project_symlinks.py
+++ b/readthedocs/rtd_tests/tests/test_project_symlinks.py
@@ -14,7 +14,7 @@
 
 from readthedocs.builds.models import Version
 from readthedocs.projects.models import Project, Domain
-from readthedocs.projects.tasks import symlink_project, remove_orphan_symlinks
+from readthedocs.projects.tasks import broadcast_remove_orphan_symlinks, remove_orphan_symlinks, symlink_project
 from readthedocs.core.symlink import PublicSymlink, PrivateSymlink
 
 
@@ -238,6 +238,16 @@ def test_symlink_remove_orphan_symlinks(self):
 
         self.assertFilesystem(filesystem)
 
+    def test_broadcast_remove_orphan_symlinks(self):
+        """Broadcast orphan symlinks is called with the proper attributes."""
+        with mock.patch('readthedocs.projects.tasks.broadcast') as broadcast:
+            broadcast_remove_orphan_symlinks()
+
+        broadcast.assert_called_with(
+            type='web',
+            task=remove_orphan_symlinks,
+            args=[],
+        )
 
     def test_symlink_cname_dont_link_missing_domains(self):
         """Domains should be relinked after deletion"""