Repository: inventree/InvenTree

Issue:
[BUG] Inventree fiddles with files directly rather than using Django Storage api
**Describe the bug**
Django provides an abstract [storage api](https://docs.djangoproject.com/en/4.0/ref/files/storage/#django.core.files.storage.get_storage_class) that should be used for all file accesses so that the server can be run from places that do not have a local filesystem and/or shared filesystem to store the media files.

**Steps to Reproduce**

Steps to reproduce the behavior:
Run Inventree API server on a node with a readonly filesystem and you will crash out in [label/app.py](https://github.com/inventree/InvenTree/blob/dd3ac58f38e0f27f18b48dcded7bc84043539d5f/InvenTree/label/apps.py#L70) because it cannot create the folders, no matter what the DEFAULT_STORAGE_CLASS is set to.

**Expected behavior**
I expect that inventree should honour the DEFAULT_STORAGE_CLASS for writing and reading its media files.

**Deployment Method**
- [x] Docker
- [x] Kubernetes
- [ ] Bare Metal

**Version Information**
Application fails to start up.  This using the master branch of the main Inventree repository.



Discussion:
@nwns this is a known side effect of the current implementation of file access.
The storage API can help with this but introduces a number of possible bugs/security issues with different storage providers as some of them (boto for example) do not respect the originally intended access rights.
We have to rewrite at least 30 code blocks, possible more, to get full compliance with the API and introduce extra security measures so that users do not just expose all their part files, photos, test results and reports to the world. Especially with cloud deployments like Kubernetes in mind.

So this is a rather sensitive task - contributions are very welcome. I would love to get to this but I am slammed with work rn.
When implementing this we should also document and test a few possible storage classes and recommended settings for those. Especially block storage services like Amazon S3 can be dangerous as they enable fast misconfiguration and could lead to circumvention of our privileges system.
The changes for running in k8s will come with at least one example of a S3 like API for testing.

More importantly, we need to stop adding new code that directly fiddles with the filesystem and uses the Storages api, even if the only production supported storage is local files for now.
@nwns I will have to disagree here. Current standard in the codebase is to use pythons built-in file-APIs (fiddle as you call it). Using 2 different APIs for the same thing sounds like a great source for strange, hard to reproduce, special case bugs.
If we switch to Django's storage API this needs to be for the whole codebase and not a bit-per-bit situation as there need to be docs etc. connected to it. The storage API can behave strange with symlinks on windows, some user might have to change their storage setups after the switch.
@matmair 
>Using 2 different APIs for the same thing sounds like a great source for strange, hard to reproduce, special case bugs.

Agreed.  However, I will also note, from what I can tell, Inventree hardcodes the default file storage so until that can be changed, it should be fine.  Unless, someone is maintaining their own settings.py (which I used to do, I would *not* recommend), but if they are, they are also advanced enough to know how this works.

Who is using symlinks? How are they using symlinks?  For the media files?  I searched the code and the only symlink stuff I can find is in various libraries, all of which should know if what they are trying to do will not work on windows and do a copy.  In particular django itself can use symlinks for collect static, but this is about media files.

To be clear, I am not suggesting that Inventree should change the default location for media files or the default behaviour, they are both fine when Inventree is run on a single machine with a persistent writable disk.
@nwns symlinks are often used in big MSFT environments when deploying with IIS. I would never recommend using that but some enterprises have strict rules on what can be deployed so they have to use IIS.
Overriding the settings is done a lot from what I hear. One of the reasons a lot of ppl. keep using old releases for a long time.
I find it a bad idea to use 2 APIs, feel free to submit PRs how you like. There are no concrete coding styles for this org.
@SchrodingersGat I look at this while handling #2861 and my suggestion would be to move:
- plugins.txt
- config.yaml
- locale_stats.json

to the root media folder

Resolving patch:
diff --git a/InvenTree/InvenTree/files.py b/InvenTree/InvenTree/files.py
new file mode 100644
--- /dev/null
+++ b/InvenTree/InvenTree/files.py
@@ -0,0 +1,8 @@
+"""Helpers for file handling in InvenTree."""
+
+from pathlib import Path
+
+from django.conf import settings
+
+TEMPLATES_DIR = Path(__file__).parent.parent
+MEDIA_STORAGE_DIR = settings.MEDIA_ROOT
diff --git a/InvenTree/InvenTree/helpers.py b/InvenTree/InvenTree/helpers.py
--- a/InvenTree/InvenTree/helpers.py
+++ b/InvenTree/InvenTree/helpers.py
@@ -9,14 +9,15 @@
 import os.path
 import re
 from decimal import Decimal, InvalidOperation
-from typing import TypeVar
+from pathlib import Path
+from typing import TypeVar, Union
 from wsgiref.util import FileWrapper
 
 import django.utils.timezone as timezone
 from django.conf import settings
 from django.contrib.staticfiles.storage import StaticFilesStorage
 from django.core.exceptions import FieldError, ValidationError
-from django.core.files.storage import default_storage
+from django.core.files.storage import Storage, default_storage
 from django.http import StreamingHttpResponse
 from django.utils.translation import gettext_lazy as _
 
@@ -861,9 +862,14 @@ def hash_barcode(barcode_data):
     return str(hash.hexdigest())
 
 
-def hash_file(filename: str):
+def hash_file(filename: Union[str, Path], storage: Union[Storage, None] = None):
     """Return the MD5 hash of a file."""
-    return hashlib.md5(open(filename, 'rb').read()).hexdigest()
+    content = (
+        open(filename, 'rb').read()
+        if storage is None
+        else storage.open(str(filename), 'rb').read()
+    )
+    return hashlib.md5(content).hexdigest()
 
 
 def server_timezone() -> str:
diff --git a/InvenTree/generic/templating/apps.py b/InvenTree/generic/templating/apps.py
--- a/InvenTree/generic/templating/apps.py
+++ b/InvenTree/generic/templating/apps.py
@@ -1,11 +1,9 @@
 """Shared templating code."""
 
 import logging
-import os
 import warnings
 from pathlib import Path
 
-from django.conf import settings
 from django.core.exceptions import AppRegistryNotReady
 from django.core.files.storage import default_storage
 from django.db.utils import IntegrityError, OperationalError, ProgrammingError
@@ -18,9 +16,6 @@
 logger = logging.getLogger('inventree')
 
 
-MEDIA_STORAGE_DIR = Path(settings.MEDIA_ROOT)
-
-
 class TemplatingMixin:
     """Mixin that contains shared templating code."""
 
@@ -84,8 +79,7 @@ def create_template_dir(self, model, data):
 
         # Create root dir for templates
         src_dir = self.get_src_dir(ref_name)
-        dst_dir = MEDIA_STORAGE_DIR.joinpath(self.name, 'inventree', ref_name)
-        ensure_dir(dst_dir, default_storage)
+        ensure_dir(Path(self.name, 'inventree', ref_name), default_storage)
 
         # Copy each template across (if required)
         for entry in data:
@@ -94,29 +88,27 @@ def create_template_dir(self, model, data):
     def create_template_file(self, model, src_dir, data, ref_name):
         """Ensure a label template is in place."""
         # Destination filename
-        filename = os.path.join(self.name, 'inventree', ref_name, data['file'])
-
+        filename = Path(self.name, 'inventree', ref_name, data['file'])
         src_file = src_dir.joinpath(data['file'])
-        dst_file = MEDIA_STORAGE_DIR.joinpath(filename)
 
         do_copy = False
 
-        if not dst_file.exists():
+        if not default_storage.exists(filename):
             logger.info("%s template '%s' is not present", self.name, filename)
             do_copy = True
         else:
             # Check if the file contents are different
             src_hash = InvenTree.helpers.hash_file(src_file)
-            dst_hash = InvenTree.helpers.hash_file(dst_file)
+            dst_hash = InvenTree.helpers.hash_file(filename, default_storage)
 
             if src_hash != dst_hash:
                 logger.info("Hash differs for '%s'", filename)
                 do_copy = True
 
         if do_copy:
-            logger.info("Copying %s template '%s'", self.name, dst_file)
+            logger.info("Copying %s template '%s'", self.name, filename)
             # Ensure destination dir exists
-            dst_file.parent.mkdir(parents=True, exist_ok=True)
+            ensure_dir(filename.parent, default_storage)
 
             # Copy file
             default_storage.save(filename, src_file.open('rb'))
@@ -135,6 +127,8 @@ def create_template_file(self, model, src_dir, data, ref_name):
         logger.info("Creating entry for %s '%s'", model, data.get('name'))
 
         try:
-            model.objects.create(**self.get_new_obj_data(data, filename))
-        except Exception:
-            logger.warning("Failed to create %s '%s'", self.name, data['name'])
+            model.objects.create(**self.get_new_obj_data(data, str(filename)))
+        except Exception as _e:
+            logger.warning(
+                "Failed to create %s '%s' with error '%s'", self.name, data['name'], _e
+            )


Test patch:
diff --git a/InvenTree/report/tests.py b/InvenTree/report/tests.py
--- a/InvenTree/report/tests.py
+++ b/InvenTree/report/tests.py
@@ -3,7 +3,6 @@
 import os
 import shutil
 from io import StringIO
-from pathlib import Path
 
 from django.conf import settings
 from django.core.cache import cache
@@ -19,6 +18,7 @@
 import report.models as report_models
 from build.models import Build
 from common.models import InvenTreeSetting, InvenTreeUserSetting
+from InvenTree.files import MEDIA_STORAGE_DIR, TEMPLATES_DIR
 from InvenTree.unit_test import InvenTreeAPITestCase
 from report.templatetags import barcode as barcode_tags
 from report.templatetags import report as report_tags
@@ -249,11 +249,9 @@ def setUp(self):
 
     def copyReportTemplate(self, filename, description):
         """Copy the provided report template into the required media directory."""
-        src_dir = Path(__file__).parent.joinpath('templates', 'report')
-
+        src_dir = TEMPLATES_DIR.joinpath('report', 'templates', 'report')
         template_dir = os.path.join('report', 'inventree', self.model.getSubdir())
-
-        dst_dir = settings.MEDIA_ROOT.joinpath(template_dir)
+        dst_dir = MEDIA_STORAGE_DIR.joinpath(template_dir)
 
         if not dst_dir.exists():  # pragma: no cover
             dst_dir.mkdir(parents=True, exist_ok=True)