Repository: ResonantGeoData/ResonantGeoData

Issue:
Use memcached with large_image
Right now, large_image is using Python memory for caching per process (large_image logs `Using python for large_image caching` anytime we request tiles). This means the cache is not actually being used when serving tiles as each request comes in through a different process.

We need to configure large_image to use Memcached:

```py
large_image.config.setConfig('cache_backend', 'memcached')
```

This requires a memcached server running somewhere - we may need this in both the django and celery services. My current impression is that different services (on different nodes/machines) cannot share a cache


Discussion:
memcached can be configured on a separate machine using the config settings `cache_memcached_url` (defaults to `127.0.0.1`), `cache_memcached_username`, and `cache_memcached_password`.  This is only useful if network access to the memcached server is faster than tile access from the source files.
Thanks for the tip, @manthey! I'm curious what exactly the keys are for the cached tiles? Are the keys, the file path (and open parameters like `projection`)?

I'm wondering if we could have our web server and our worker share a memcache server for the tiles and if that's possible - I suppose this means we will have to have a predetermined path that is consistent across servers (e.g. `/tmp/rgd/file_cache/...`.
The code in Girder 3 is here: https://github.com/girder/large_image/blob/0a29523923106f079c35b6de6cf7e9e04aea4434/girder/girder_large_image/models/image_item.py#L192
Fundamentally, we know we record what tile source was sued (gdal, openslide, etc).  From that, the hash is taken from the source classes hash (based on the path and the options such as projection and style), plus a string hash based on the tile parameters.

So, yes, you'd want a consistent path across servers OR you'd need to redefine how the class hashes are generated (so it uses, for instance, the primary key from the django table instead of the path).

Resolving patch:
diff --git a/django-rgd-imagery/rgd_imagery/__init__.py b/django-rgd-imagery/rgd_imagery/__init__.py
--- a/django-rgd-imagery/rgd_imagery/__init__.py
+++ b/django-rgd-imagery/rgd_imagery/__init__.py
@@ -1,3 +1,7 @@
+import logging
+
+from django.conf import settings
+import large_image
 from pkg_resources import DistributionNotFound, get_distribution
 
 try:
@@ -5,3 +9,22 @@
 except DistributionNotFound:
     # package is not installed
     __version__ = None
+
+
+logger = logging.getLogger(__name__)
+
+# Set config options for large_image
+if hasattr(settings, 'RGD_MEMCACHED_URL') and settings.RGD_MEMCACHED_URL:
+    large_image.config.setConfig('cache_memcached_url', settings.RGD_MEMCACHED_URL)
+    if (
+        hasattr(settings, 'RGD_MEMCACHED_USERNAME')
+        and settings.RGD_MEMCACHED_USERNAME
+        and hasattr(settings, 'RGD_MEMCACHED_PASSWORD')
+        and settings.RGD_MEMCACHED_PASSWORD
+    ):
+        large_image.config.setConfig('cache_memcached_username', settings.RGD_MEMCACHED_USERNAME)
+        large_image.config.setConfig('cache_memcached_password', settings.RGD_MEMCACHED_PASSWORD)
+    large_image.config.setConfig('cache_backend', 'memcached')
+    logger.info('Configured for memcached.')
+else:
+    logger.info('Settings not properly configured for memcached.')
diff --git a/django-rgd-imagery/setup.py b/django-rgd-imagery/setup.py
--- a/django-rgd-imagery/setup.py
+++ b/django-rgd-imagery/setup.py
@@ -47,7 +47,7 @@
     install_requires=[
         'bidict',
         'django-rgd',
-        'large-image>=1.8.1.dev7',
+        'large-image[memcached]>=1.8.1.dev7',
         'large-image-source-gdal>=1.8.1.dev7',
         'large-image-source-pil>=1.8.1.dev7',
         'large-image-source-tiff>=1.8.1.dev7',
diff --git a/django-rgd/rgd/configuration/configuration.py b/django-rgd/rgd/configuration/configuration.py
--- a/django-rgd/rgd/configuration/configuration.py
+++ b/django-rgd/rgd/configuration/configuration.py
@@ -71,3 +71,6 @@ def before_binding(configuration: ComposedConfiguration) -> None:
     RGD_STAC_BROWSER_LIMIT = values.Value(default=1000)
     RGD_TEMP_DIR = values.Value(default=os.path.join(tempfile.gettempdir(), 'rgd'))
     RGD_TARGET_AVAILABLE_CACHE = values.Value(default=2)
+    RGD_MEMCACHED_USERNAME = values.Value(default=None)
+    RGD_MEMCACHED_PASSWORD = values.Value(default=None)
+    RGD_MEMCACHED_URL = values.Value(default=None)


Test patch: