Repository: fossasia/open-event-server

Issue:
Make file exports take into consideration the storage setting on the app
Right now, file exports write the exported zip to the local file system by default irrespective of what the user's storage setting (google cloud, s3, local) is. Ensure the exported zip is placed in the appropriate location.

This is especially important when we have something like a kubernetes deployment where the local-storage is volatile and isolated by default. Local storage is not sharable b/w pods by default. (In our case, celery's local storage is isolated from the web app's local storage, so, creating a shared persistent storage space that is read-writable by multiple pods is both tricky and slightly unreliable during re-deployments)


Discussion:
Adding to this issue:
Right now we are not deleting the exported zip. As the zip is being stored in a volatile storage, this wasn't seen necessary. But now as the zip will be stored on S3 or google cloud, they will have to be deleted at some point (+ 24 hours) so that they don't end up taking space. A celery scheduled job will easily do this. 

PS - I would have loved to fix this issue but alas, end semester exams this week. 
I'm working on this

Resolving patch:
diff --git a/app/api/helpers/export_helpers.py b/app/api/helpers/export_helpers.py
--- a/app/api/helpers/export_helpers.py
+++ b/app/api/helpers/export_helpers.py
@@ -23,6 +23,8 @@
 from ..speakers import DAO as SpeakerDAO, SPEAKER
 from ..sponsors import DAO as SponsorDAO, SPONSOR
 from ..tracks import DAO as TrackDAO, TRACK
+from app.helpers.storage import upload, UPLOAD_PATHS, UploadedFile
+from app.settings import get_settings
 
 # DELETE FIELDS
 # All fields to be deleted go here
@@ -207,7 +209,17 @@ def export_event_json(event_id, settings):
     fp.close()
     # make zip
     shutil.make_archive(dir_path, 'zip', dir_path)
-    return dir_path + '.zip'
+    dir_path = dir_path + ".zip"
+
+    storage_path = UPLOAD_PATHS['exports']['zip'].format(
+        event_id = event_id
+    )
+    uploaded_file = UploadedFile(dir_path, dir_path.rsplit('/', 1)[1])
+    storage_url = upload(uploaded_file, storage_path)
+
+    if get_settings()['storage_place'] != "s3" or get_settings()['storage_place'] != 'gs':
+        storage_url = app.config['BASE_DIR'] + storage_url.replace("/serve_","/")
+    return storage_url
 
 
 # HELPERS
diff --git a/app/api/helpers/tasks.py b/app/api/helpers/tasks.py
--- a/app/api/helpers/tasks.py
+++ b/app/api/helpers/tasks.py
@@ -16,6 +16,7 @@
 from import_helpers import update_import_job, send_import_mail
 from app.api.exports import event_export_task_base
 from app.api.imports import import_event_task_base
+from app.settings import get_settings
 
 
 @celery.task(base=RequestContextTask, name='import.event', bind=True, throws=(BaseError,))
@@ -46,10 +47,15 @@ def export_event_task(self, event_id, settings):
         logging.info('Exporting started')
         path = event_export_task_base(event_id, settings)
         # task_id = self.request.id.__str__()  # str(async result)
-        result = {
-            'download_url': url_for(
+        if get_settings()['storage_place'] == 'local' or get_settings()['storage_place'] == None:
+            download_url = url_for(
                 'api.exports_export_download', event_id=event_id, path=path
             )
+        else:
+            download_url = path
+
+        result = {
+            'download_url': download_url
         }
     except BaseError as e:
         result = {'__error': True, 'result': e.to_dict()}
diff --git a/app/helpers/storage.py b/app/helpers/storage.py
--- a/app/helpers/storage.py
+++ b/app/helpers/storage.py
@@ -51,6 +51,9 @@
     'temp': {
         'event': 'events/temp/{uuid}',
         'image': 'temp/images/{uuid}'
+    },
+    'exports': {
+        'zip': 'exports/{event_id}'
     }
 }
 


Test patch: