Repository: hydroshare/hydroshare

Issue:
update signal processor to prevent unnecessary recursion
Related to #5015 
I just dropped a breakpoint in [handle_save](https://github.com/hydroshare/hydroshare/blob/2.4/hs_core/hydro_realtime_signal_processor.py#L20) and was surprised that **updating the title of a resource**, a simple change, results in 6 calls to `handle_save`!

1. <Title: Untitled Resourceaaaa>
2. <BaseResource: Untitled Resourceaaaa>
3. <CompositeResource: Untitled Resourceaaaaa>
4. <Date: modified 2023-03-21T13:15:33.646070+00:00>
5. <BaseResource: Untitled Resourceaaaaa>
6. <Variable: Variable object (920)>

1 and 4 are sensical enough. I can potentially see why we need 2 to update the resource index even though we update the Title metadata in 1. I don't understand why we need 3, but I can be convinced easily. 5 is perplexing...seems like a duplicate -- for a simple title change we index the BaseResource to  _twice!?_ oh but I do see that `handle_save` calls itself recursively so that prob what's going on. And then 6 I believe this is an hs_tracking Variable which doesn't get indexed so I guess it's okay.

I feel like some of these calls can obviously be async as @sblack-usu suggested. But might we run into race conditions etc if we are saving the updated date (4) and the BaseResource (2, 5)? Think the answer is "NO" because we are just talking about updating the indexes async, so I would think there wouldn't be a problem (vs for instance, actually saving to the Django db).

_Originally posted by @devincowan in https://github.com/hydroshare/hydroshare/issues/5015#issuecomment-1477839803_
            
Seems like we might want to revise our signal processor, [custom haystack processor](https://django-haystack.readthedocs.io/en/v2.4.1/signal_processors.html#custom-signalprocessors)
We are thinking there could be significant performance gain if we trigger (async?) indexing not for all saved objects but just for `Raccess` and `Date` metadata objects with `type=modified`


Discussion:
Also worth noting that a simple **view** (not edit/save) of the resource landing page triggers ~5 calls the the `handle_save` method of our signal processor 👎 
I'm excited to see the performance gains from this!  It looks promising.
@devincowan 

> Also worth noting that a simple view (not edit/save) of the resource landing page triggers ~5 calls the the handle_save method of our signal processor

The fix I made for #5015 should result in no indexing for resource view landing page. 

Resolving patch:
diff --git a/hs_core/hydro_realtime_signal_processor.py b/hs_core/hydro_realtime_signal_processor.py
--- a/hs_core/hydro_realtime_signal_processor.py
+++ b/hs_core/hydro_realtime_signal_processor.py
@@ -1,108 +1,71 @@
 import logging
 
 from django.conf import settings
+from django.db import models
+from hs_core.models import Date, BaseResource
+from hs_access_control.models import ResourceAccess
 from haystack.exceptions import NotHandled
-from haystack.signals import RealtimeSignalProcessor
+from haystack.signals import BaseSignalProcessor
 
 logger = logging.getLogger(__name__)
 
 
-class HydroRealtimeSignalProcessor(RealtimeSignalProcessor):
-
+class HydroRealtimeSignalProcessor(BaseSignalProcessor):
     """
-    Customized for the fact that all indexed resources are subclasses of BaseResource.
     Notes:
-    1. RealtimeSignalProcessor already plumbs in all class updates. We might want to be more specific.
-    2. The class sent to this is a subclass of BaseResource, or another class.
-    3. Thus, we want to capture cases in which it is an appropriate instance, and respond.
+    1. We assume everytime metadata is updated the modified datetime is updated
+    2. ResourceAccess does not update the modified datetime (it is not scientific metadata)
     """
 
-    def handle_save(self, sender, instance, **kwargs):
-        """
-        Given an individual model instance, determine which backends the
-        update should be sent to & update the object on those backends.
-        """
-
-        from hs_core.models import BaseResource, CoreMetaData, AbstractMetaDataElement
-        from hs_access_control.models import ResourceAccess
-        from hs_file_types.models import AbstractFileMetaData
-        from django.contrib.postgres.fields import HStoreField
-
-        if getattr(settings, "DISABLE_HAYSTACK", False):
-            # haystack has been set to disable state (during test run)
-            # a resource can be indexed as part of test run by setting one of the keywords as 'INDEX-FOR-TESTING'
-            if isinstance(instance, BaseResource) and not instance.metadata.subjects.filter(
-                    value="INDEX-FOR-TESTING").exists():
-                # no need to index
-                return
+    def setup(self):
+        if not getattr(settings, "DISABLE_HAYSTACK", False):
+            models.signals.post_save.connect(self.handle_update, sender=Date)
+            models.signals.post_save.connect(self.handle_access, sender=ResourceAccess)
 
-        if isinstance(instance, BaseResource):
-            if hasattr(instance, 'raccess') and hasattr(instance, 'metadata'):
-                # work around for failure of super(BaseResource, instance) to work properly.
-                # this always succeeds because this is a post-save object action.
-                newbase = BaseResource.objects.get(pk=instance.pk)
-                newsender = BaseResource
-                using_backends = self.connection_router.for_write(instance=newbase)
-                for using in using_backends:
-                    # if object is public/discoverable or becoming public/discoverable, index it
-                    # test whether the object should be exposed.
-                    if instance.show_in_discover:
-                        try:
-                            index = self.connections[using].get_unified_index().get_index(newsender)
-                            index.update_object(newbase, using=using)
-                        except NotHandled:
-                            logger.exception("Failure: changes to %s with short_id %s not added to Solr Index.",
-                                             str(type(instance)), newbase.short_id)
+    def teardown(self):
+        if not getattr(settings, "DISABLE_HAYSTACK", False):
+            models.signals.post_save.disconnect(self.handle_update, sender=Date)
+            models.signals.post_save.disconnect(self.handle_access, sender=ResourceAccess)
 
-                    # if object is private or becoming private, delete from index
-                    else:  # not to be shown in discover
-                        try:
-                            index = self.connections[using].get_unified_index().get_index(newsender)
-                            index.remove_object(newbase, using=using)
-                        except NotHandled:
-                            logger.exception("Failure: delete of %s with short_id %s failed.",
-                                             str(type(instance)), newbase.short_id)
+    def handle_update(self, sender, instance, **kwargs):
+        try:
+            # resolve the BaseResource corresponding to the metadata element.
+            newbase = instance.metadata.resource
+            index_resource(self, newbase)
+        except Exception as e:
+            logger.exception("{} exception: {}".format(type(instance), str(e)))
 
-        elif isinstance(instance, ResourceAccess):
-            # automatically a BaseResource; just call the routine on it.
-            try:
-                newbase = instance.resource
-                self.handle_save(BaseResource, newbase)
-            except Exception as e:
-                logger.exception("{} exception: {}".format(type(instance), str(e)))
+    def handle_access(self, sender, instance, **kwargs):
+        try:
+            newbase = instance.resource
+            index_resource(self, newbase)
+        except Exception as e:
+            logger.exception("{} exception: {}".format(type(instance), str(e)))
 
-        elif isinstance(instance, CoreMetaData):
-            try:
-                newbase = instance.resource
-                self.handle_save(BaseResource, newbase)
-            except Exception as e:
-                logger.exception("{} exception: {}".format(type(instance), str(e)))
 
-        elif isinstance(instance, AbstractMetaDataElement):
-            if isinstance(instance.metadata, AbstractFileMetaData):
+def index_resource(signal_processor, instance: BaseResource):
+    if hasattr(instance, 'raccess') and hasattr(instance, 'metadata'):
+        # work around for failure of super(BaseResource, instance) to work properly.
+        # this always succeeds because this is a post-save object action.
+        newbase = BaseResource.objects.get(pk=instance.pk)
+        newsender = BaseResource
+        using_backends = signal_processor.connection_router.for_write(instance=newbase)
+        for using in using_backends:
+            # if object is public/discoverable or becoming public/discoverable, index it
+            # test whether the object should be exposed.
+            if instance.show_in_discover:
                 try:
-                    # resolve the BaseResource corresponding to the metadata element in a logical logical.
-                    newbase = instance.metadata.logical_file.resource
-                    self.handle_save(BaseResource, newbase)
-                except Exception as e:
-                    logger.exception("{} exception: {}".format(type(instance), str(e)))
-            else:
-                try:
-                    # resolve the BaseResource corresponding to the metadata element.
-                    newbase = instance.metadata.resource
-                    self.handle_save(BaseResource, newbase)
-                except Exception as e:
-                    logger.exception("{} exception: {}".format(type(instance), str(e)))
-
-        elif isinstance(instance, HStoreField):
-            try:
-                newbase = BaseResource.objects.get(extra_metadata=instance)
-                self.handle_save(BaseResource, newbase)
-            except Exception as e:
-                logger.exception("{} exception: {}".format(type(instance), str(e)))
+                    index = signal_processor.connections[using].get_unified_index().get_index(newsender)
+                    index.update_object(newbase, using=using)
+                except NotHandled:
+                    logger.exception("Failure: changes to %s with short_id %s not added to Solr Index.",
+                                     str(type(instance)), newbase.short_id)
 
-    def handle_delete(self, sender, instance, **kwargs):
-        """
-        Ignore delete events as this is accomplished separately.
-        """
-        pass
+            # if object is private or becoming private, delete from index
+            else:  # not to be shown in discover
+                try:
+                    index = signal_processor.connections[using].get_unified_index().get_index(newsender)
+                    index.remove_object(newbase, using=using)
+                except NotHandled:
+                    logger.exception("Failure: delete of %s with short_id %s failed.",
+                                     str(type(instance)), newbase.short_id)
diff --git a/hydroshare/settings.py b/hydroshare/settings.py
--- a/hydroshare/settings.py
+++ b/hydroshare/settings.py
@@ -782,7 +782,6 @@ def _(s): return s # noqa
     import logging
 
     logging.disable(logging.CRITICAL)
-    DISABLE_HAYSTACK = True
     PASSWORD_HASHERS = [
         'django.contrib.auth.hashers.MD5PasswordHasher',
     ]


Test patch: