Repository: readthedocs/readthedocs.org

Issue:
Search index not updated after renaming/moving of a page
#5845 renames [APIv3 Design Document](https://docs.readthedocs.io/en/latest/development/design/apiv3.html) from `Design of APIv3` to `APIv3 Design Document` but the search results still contains both results.

Related link: https://docs.readthedocs.io/en/latest/index.html?rtd_search=apiv3

![Screenshot from 2019-08-13 21-23-29](https://user-images.githubusercontent.com/29149191/62956950-7816ea80-be11-11e9-8338-3b873db0065d.png)

Remove files from cloud storage when version is wiped.
Fixes #6069 


Discussion:
There is this error on sentry, maybe related to this issue

```
update or delete on table "projects_importedfile" violates foreign key constraint "sphinx_domains_sphin_html_file_id_a0307e9f_fk_projects_" on table "sphinx_domains_sphinxdomain"
```

It happens here

https://github.com/readthedocs/readthedocs.org/blob/e53af663c0996303c249743123b92ae886c1734f/readthedocs/projects/tasks.py#L1640-L1647

https://sentry.io/organizations/read-the-docs/issues/1144425753/?project=148442&query=is%3Aunresolved&statsPeriod=14d
Not sure if this is related, but for https://docs.fluxcd.io, we also notice that renamed files and files removed from git aren't removed from the index.

E.g. first result for this search: https://docs.fluxcd.io/en/latest/search.html?q=ignore&check_keywords=yes&area=default
@hiddeco 
I think wiping the version and then rebuilding the docs should fix the problem.
You can find the steps for it here: https://docs.readthedocs.io/en/stable/guides/wipe-environment.html
@dojutsu-user already did this before I commented, without any effect.
Thanks @stsewd for flagging this still-open issue. We're observing this on the great expectations docs (https://great-expectations.readthedocs.io/en/latest/). Searching for a moved page shows both the new (working) and broken link. I have tried wiping and rebuilding to no effect.
Should we add a FAQ question for this?
Something like -- `How can I remove old results from the search results?`

Edit: Added the FAQ.
> Should we add a FAQ question for this?

Not sure about that, if the files aren't refreshed that seems like a bug from our side.
@humitos : what do you think should be the way to trigger the removal from cloud storage? As someone following this issue from outside, the primary issue for me is that I have a bad search index in cloud storage and no way at all to fix it.
I'm not proposing a solution for the issue. I'm not sure how it should be.

I'm saying that the proposed solution in this PR could introduce a behavior (a bug, to me) that we don't want.

I may be wrong, though.
@humitos I think you may be right. Then I think the problem is something similar to what we had in .com with the indexes. We are reading the json objects from the wrong directory.
And also, maybe we are not cleaning the old files from storage before a new build as we do with the file based storage
I think that @davidfischer can provide more info here.
> I think that @davidfischer can provide more info here.

Credit to @stsewd but I think the issue is that with filesystem backed storage, we were running rsync which both copied new/updated files **and deleted removed files**. The current storage system doesn't do that with cloud storage.

Resolving patch:
diff --git a/readthedocs/builds/storage.py b/readthedocs/builds/storage.py
--- a/readthedocs/builds/storage.py
+++ b/readthedocs/builds/storage.py
@@ -4,8 +4,7 @@
 from django.conf import settings
 from django.core.exceptions import SuspiciousFileOperation
 from django.core.files.storage import FileSystemStorage
-from storages.utils import safe_join, get_available_overwrite_name
-
+from storages.utils import get_available_overwrite_name, safe_join
 
 log = logging.getLogger(__name__)
 
@@ -88,6 +87,48 @@ def copy_directory(self, source, destination):
                 with filepath.open('rb') as fd:
                     self.save(sub_destination, fd)
 
+    def sync_directory(self, source, destination):
+        """
+        Sync a directory recursively to storage.
+
+        Overwrites files in remote storage with files from ``source`` (no timstamp/hash checking).
+        Removes files and folders in remote storage that are not present in ``source``.
+
+        :param source: the source path on the local disk
+        :param destination: the destination path in storage
+        """
+        if destination in ('', '/'):
+            raise SuspiciousFileOperation('Syncing all storage cannot be right')
+
+        log.debug(
+            'Syncing to media storage. source=%s destination=%s',
+            source, destination,
+        )
+        source = Path(source)
+        copied_files = set()
+        copied_dirs = set()
+        for filepath in source.iterdir():
+            sub_destination = self.join(destination, filepath.name)
+            if filepath.is_dir():
+                # Recursively sync the subdirectory
+                self.sync_directory(filepath, sub_destination)
+                copied_dirs.add(filepath.name)
+            elif filepath.is_file():
+                with filepath.open('rb') as fd:
+                    self.save(sub_destination, fd)
+                copied_files.add(filepath.name)
+
+        # Remove files that are not present in ``source``
+        dest_folders, dest_files = self.listdir(self._dirpath(destination))
+        for folder in dest_folders:
+            if folder not in copied_dirs:
+                self.delete_directory(self.join(destination, folder))
+        for filename in dest_files:
+            if filename not in copied_files:
+                filepath = self.join(destination, filename)
+                log.debug('Deleting file from media storage. file=%s', filepath)
+                self.delete(filepath)
+
     def join(self, directory, filepath):
         return safe_join(directory, filepath)
 
diff --git a/readthedocs/projects/tasks.py b/readthedocs/projects/tasks.py
--- a/readthedocs/projects/tasks.py
+++ b/readthedocs/projects/tasks.py
@@ -847,7 +847,7 @@ def store_build_artifacts(
                 },
             )
             try:
-                storage.copy_directory(from_path, to_path)
+                storage.sync_directory(from_path, to_path)
             except Exception:
                 # Ideally this should just be an IOError
                 # but some storage backends unfortunately throw other errors


Test patch:
diff --git a/readthedocs/rtd_tests/tests/test_build_storage.py b/readthedocs/rtd_tests/tests/test_build_storage.py
--- a/readthedocs/rtd_tests/tests/test_build_storage.py
+++ b/readthedocs/rtd_tests/tests/test_build_storage.py
@@ -18,6 +18,25 @@ def setUp(self):
     def tearDown(self):
         shutil.rmtree(self.test_media_dir, ignore_errors=True)
 
+    def assertFileTree(self, source, tree):
+        """
+        Recursively check that ``source`` from storage has the same file tree as ``tree``.
+
+        :param source: source path in storage
+        :param tree: a list of strings representing files
+                     or tuples (string, list) representing directories.
+        """
+        dirs_tree = [e for e in tree if not isinstance(e, str)]
+
+        dirs, files = self.storage.listdir(source)
+        expected_dirs = [e[0] for e in dirs_tree]
+        expected_files = [e for e in tree if isinstance(e, str)]
+        self.assertCountEqual(dirs, expected_dirs)
+        self.assertCountEqual(files, expected_files)
+
+        for folder, files in dirs_tree:
+            self.assertFileTree(self.storage.join(source, folder), files)
+
     def test_copy_directory(self):
         self.assertFalse(self.storage.exists('files/test.html'))
 
@@ -27,6 +46,39 @@ def test_copy_directory(self):
         self.assertTrue(self.storage.exists('files/api.fjson'))
         self.assertTrue(self.storage.exists('files/api/index.html'))
 
+    def test_sync_directory(self):
+        tmp_files_dir = os.path.join(tempfile.mkdtemp(), 'files')
+        shutil.copytree(files_dir, tmp_files_dir)
+        storage_dir = 'files'
+
+        tree = [
+            ('api', ['index.html']),
+            'api.fjson',
+            'conf.py',
+            'test.html',
+        ]
+        self.storage.sync_directory(tmp_files_dir, storage_dir)
+        self.assertFileTree(storage_dir, tree)
+
+        tree = [
+            ('api', ['index.html']),
+            'conf.py',
+            'test.html',
+        ]
+        os.remove(os.path.join(tmp_files_dir, 'api.fjson'))
+        self.storage.sync_directory(tmp_files_dir, storage_dir)
+        self.assertFileTree(storage_dir, tree)
+
+        tree = [
+            # Cloud storage generally doesn't consider empty directories to exist
+            ('api', []),
+            'conf.py',
+            'test.html',
+        ]
+        shutil.rmtree(os.path.join(tmp_files_dir, 'api'))
+        self.storage.sync_directory(tmp_files_dir, storage_dir)
+        self.assertFileTree(storage_dir, tree)
+
     def test_delete_directory(self):
         self.storage.copy_directory(files_dir, 'files')
         dirs, files = self.storage.listdir('files')