torchbox/wagtail — pull request #714 by kaedroho (2014-10-17 12:04:02+00:00)
Title: Search: Added new deduplication method Fixes #710

One problem wagtailsearch faces is the fact that pages could be indexed as a ``Page`` object or their most specific class. We always want to only index the most specific version but the challenging is finding that quickly.

Currently, this is solved by loading every object that is in the database into memory and searching manually for duplicates. This has the following disadvantages:
- Very slow for large websites
- Doesn't work well with the signal handlers. We have to trust that Django only calls the ``post_save`` signal for the most specific object to prevent duplicates, but it doesn't do this in certain situations (such as unpublishing a page which saves only the ``Page`` instance). ``update_index`` needs to be run periodically to get rid of these duplicates.

This pull request changes the deduplication method to solve the above problems. The new method requires the developer to add a ``get_indexed_objects`` classmethod to their model. The developer then needs to make sure that this only returns a queryset of objects to return for that particular class and not subclasses (eg, ``Page.get_indexed_objects`` will only return pages that do not have a specific type, such as the root and the default homepage).

This would be used by ``update_index`` to find a list of objects to index. The signal handlers will also check that changed object exists in its ``indexed_objects`` before doing any save/delete operations.

Advantages:
- Only objects that are needed are fetched from the database
- Much fewer objects are kept in memory at a time
- Much faster
- Signal handlers will no longer create duplicates
Disadvantages:
- Developers would need to add a ``get_indexed_objects`` method
- If a ``Page`` gets saved directly, it won't be updated in the search index (as the signal will be fired on the ``Page`` object which will get blocked)