GeoNode/geonode — pull request #1562 by jj0hns0n (2014-06-27 18:29:07+00:00)
Title: Hook up Haystack

This PR actually hooks up haystack search when enabled in settings.py. Currently in master there are the settings entries for haystack, but its not actually connected anywhere. This PR remedies that with a minimal haystack implementation.

Things to NOTE: The API for the haystack endpoints is now /api/base/search/ and uses the 'q' querystring arg vs title__contains. This is setup to use the right values and URL depending on whether HAYSTACK_SEARCH is True or not.

@mbertrand and I have done some initial testing using remote services and a few thousand layers and found that the filter(oid__in=filter_set) used here https://github.com/jj0hns0n/geonode/blob/fbc2f3ea25bc944835a3e83e214cddafdcc98793/geonode/api/resourcebase_api.py#L117 is very inefficient with lots of layers and in fact requires you to change the maximum number of clauses when used with elasticsearch if the total number of layers is > 1024. With our test dataset, and all layers public, this query can take 4-5 seconds. The culprit is passing that long list to haystack rather than the actual orm or perms queries themselves. We should find an alternate way of handling this. Matt will work on converting the orm filters to simply haystack filters, but I don't currently see another way to implement the permissions filters. I think it makes sense to talk to the haystack folks and see if they have some idea how to make guardian and haystack work together in some more efficient way than this.

This branch and functionality can certainly use more improvement especially in downstream projects like worldmap that intend to leverage remote services to have many thousands of layers, but as Im running out of time to work on it, I believe it is better to get it into master in its current functional state and continue improving in master.

@ingenieroariel it would be great to hook this functionality up to your test gist and figure out how we can get it to perform as fast as is absolutely possible even for geonodes with 100s of 1000s of layers.