Repository: ResonantGeoData/ResonantGeoData

Issue:
Celery Docker service runs in privileged mode
See https://github.com/ResonantGeoData/ResonantGeoData/blob/72b3d4085cc5700d0ad5556f31b7eb96ed2d3b8a/docker-compose.override.yml#L33

Can we use libfuse without running in a privileged mode?


Discussion:
Further, I would like to use this work as a chance to make sure the CI checks are testing the usage of FUSE and also running the backup solution if FUSE is not available.
A slightly lesser-privileged method is to give the container `CAP_SYS_ADMIN` capabilities and access to `/dev/fuse`:
```
--cap-add SYS_ADMIN --device /dev/fuse
```

This is described in the [docs (ctrl-f "fuse")](https://docs.docker.com/engine/reference/run/). This should be somewhat more portable since it limits the scope of devices available to the container.

However, it's important to consider that FUSE is ultimately a kernel-level feature and Docker works with [namespaces](https://en.wikipedia.org/wiki/Linux_namespaces). Therefore, Docker containers that use FUSE may behave differently depending on the host system. With that said, [it seems like Linux 4.18+ can support mounting FUSE without additional privileges](https://lists.linuxfoundation.org/pipermail/containers/2018-June/039059.html). It might be interesting to explore that.

The use of FUSE seems like the best solution to efficiently handling remote files with GDAL. I think a responsible way to support this feature, though, is to define a minimum kernel version and capabilities. This has ramifications for deployment where Docker containers are typically expected to be strictly run in userspace.

Where do we expect to deploy this application? For each site, the following could be answered:

- What version of Linux do they run?
- What kernel modules are loaded?
- What Linux capabilities do they provide to containers?

It could be that the FUSE kernel module isn't loaded. In that case, we can't use FUSE at all.

It could be the case that all versions of Linux on the hosts are greater than Linux 4.18. In that case, it makes sense to try to mount FUSE with lesser privileges.

It could be the case that the Linux version is hard to gauge, but they all support `CAP_SYS_ADMIN` and access to `/dev/fuse`. In that case, using those capabilities is the clearest path forward.

Resolving patch:
diff --git a/django-rgd/rgd/utility.py b/django-rgd/rgd/utility.py
--- a/django-rgd/rgd/utility.py
+++ b/django-rgd/rgd/utility.py
@@ -168,10 +168,8 @@ def url_file_to_fuse_path(url: str) -> Path:
     # Assumes `precheck_fuse` was verified prior
     # See https://github.com/ResonantGeoData/ResonantGeoData/issues/237
     parsed = urlparse(url)
-    if parsed.scheme == 'https':
-        fuse_path = url.replace('https://', '/tmp/rgd/https/') + '..'
-    elif parsed.scheme == 'http':
-        fuse_path = url.replace('http://', '/tmp/rgd/http/') + '..'
+    if parsed.scheme in ['https', 'http']:
+        fuse_path = url.replace('https://', '/tmp/rgd/http/') + '..'
     elif Path('/tmp/rgd/s3/').exists() and parsed.scheme == 's3':
         fuse_path = url.replace('s3://', '/tmp/rgd/s3/') + '..'
     else:


Test patch:
diff --git a/django-rgd/tests/test_fuse.py b/django-rgd/tests/test_fuse.py
new file mode 100644
--- /dev/null
+++ b/django-rgd/tests/test_fuse.py
@@ -0,0 +1,32 @@
+from pathlib import Path
+
+import magic
+import pytest
+from rgd import utility
+from rgd.datastore import datastore
+
+SKIP_FUSE = True
+try:
+    import simple_httpfs  # noqa
+
+    assert Path('/tmp/rgd/http').exists()
+    SKIP_FUSE = False
+except (ImportError, EnvironmentError, AssertionError):
+    pass
+
+FILENAME = 'stars.png'
+
+
+@pytest.fixture
+def simple_url():
+    return datastore.get_url(FILENAME)
+
+
+@pytest.mark.skipif(SKIP_FUSE, reason='FUSE is not installed/configured.')
+@pytest.mark.django_db(transaction=True)
+def test_fuse_mount(simple_url):
+    assert utility.precheck_fuse(simple_url)
+    path = utility.url_file_to_fuse_path(simple_url)
+    assert path.exists()
+    mime = magic.Magic(mime=True)
+    assert mime.from_file(path) == 'image/png'