Repository: geoadmin/service-stac
Pull request #551: PB-1515 : add big asset upload to the collection asset
Author: login: sami-nouidri-swisstopo
Labels: name: bug
Created: 2025-03-25T08:42:58Z | Merged: 2025-03-31T06:50:13Z

This adds the big asset upload to collection assets. As discussed with @boecklic, we have added collection assets to v0.9 in order to mitigate potential API Gateway issues.

For testing, I uploaded files to existing collection assets, and was able to retrieve from the s3 bucket successfully, which indicates the upload went through (locally and on DEV). E2E tests are also successful.

Note : this needs to be rebased on this [PR](https://github.com/geoadmin/service-stac/pull/545) before merging

- [x] E2E Successful


Diff:
diff --git a/app/stac_api/admin.py b/app/stac_api/admin.py
index 64693a39..b510a50a 100644
--- a/app/stac_api/admin.py
+++ b/app/stac_api/admin.py
@@ -363,6 +363,71 @@ def save_model(self, request, obj, form, change):
         return super().save_model(request, obj, form, change)
 
 
+class AssetUploadAdminMixin:
+    upload_template_name = "uploadtemplate.html"
+    url_suffix = "_upload"
+
+    def get_urls(self):
+        urls = super().get_urls()
+        my_urls = [
+            path(
+                "<path:object_id>/change/upload/",
+                self.admin_site.admin_view(self.upload_view),
+                name=f'{self.model._meta.app_label}_{self.model._meta.model_name}{self.url_suffix}',
+            )
+        ]
+        return my_urls + urls
+
+    def upload_view(self, request, object_id, extra_context=None):
+        model = self.model
+        obj = self.get_object(request, unquote(object_id))
+        if obj is None:
+            return self._get_obj_does_not_exist_redirect(request, model._meta, object_id)
+
+        context = dict(
+            # Include common variables for rendering the admin template.
+            self.admin_site.each_context(request),
+            # Anything else you want in the context...
+            csrf_token=request.META.get('CSRF_COOKIE'),
+            asset_name=obj.name,
+            collection_name=obj.get_collection(),
+        )
+
+        if hasattr(obj, 'item'):
+            context['item_name'] = obj.item.name
+
+        return TemplateResponse(request, self.upload_template_name, context)
+
+    def change_view(self, request, object_id, form_url='', extra_context=None):
+        obj = self.model.objects.\
+            filter(id=request.resolver_match.kwargs['object_id']).first()
+
+        if getattr(obj, "is_external", False):  #check if the object is external
+            return super().change_view(request, object_id, form_url)
+
+        extra_context = extra_context or {}
+        property_upload_url = reverse(
+            f'admin:{self.model._meta.app_label}_{self.model._meta.model_name}{self.url_suffix}',
+            args=[object_id],
+        )
+        extra_context['property_upload_url'] = property_upload_url
+        return super().change_view(request, object_id, form_url, extra_context=extra_context)
+
+
+class RedirectAfterCreationMixin:
+
+    def response_add(self, request, obj, post_url_continue=None):
+        """
+        After adding a new object, redirect to the change form for the new object
+        unless the user clicked "Save and add another".
+        """
+        if "_addanother" not in request.POST:
+            request.POST = request.POST.copy()
+            request.POST["_continue"] = 1
+
+        return super().response_add(request, obj, post_url_continue)
+
+
 class CollectionAssetAdminForm(forms.ModelForm):
 
     def __init__(self, *args, **kwargs):
@@ -378,7 +443,7 @@ def __init__(self, *args, **kwargs):
 
 
 @admin.register(CollectionAsset)
-class CollectionAssetAdmin(admin.ModelAdmin):
+class CollectionAssetAdmin(AssetUploadAdminMixin, RedirectAfterCreationMixin, admin.ModelAdmin):
     form = CollectionAssetAdminForm
 
     class Media:
@@ -491,21 +556,52 @@ def displayed_file_size(self, instance):
     # That's why some fields like the name of the asset are set readonly here
     # for update operations
     def get_fieldsets(self, request, obj=None):
-        fields = super().get_fieldsets(request, obj)
+        """Build the different field sets for the admin page."""
+
+        base_fields = super().get_fieldsets(request, obj)
+
         if obj is None:
-            # In case a new Asset is added use the normal field 'collection' from model that have
-            # a help text fort the search functionality.
-            fields[0][1]['fields'] = ('name', 'collection', 'created', 'updated', 'etag')
-            return fields
-        # Otherwise if this is an update operation only display the read only fields
-        # without help text
+            return [
+                (None, {
+                    'fields': ('name', 'collection', 'created', 'updated', 'etag')
+                }),
+                ('File', {
+                    'fields': ('media_type',)
+                }),
+            ]
+
+        # Define file fields conditionally based on `allow_external_assets`
         if obj.collection.allow_external_assets:
-            fields[0][1]['fields'] = (
-                'name', 'collection_name', 'created', 'updated', 'etag', 'is_external'
+            file_fields = (
+                'is_external',
+                'file',
+                'media_type',
+                'href',
+                'checksum_multihash',
+                'update_interval',
+                'displayed_file_size'
             )
         else:
-            fields[0][1]['fields'] = ('name', 'collection_name', 'created', 'updated', 'etag')
-        return fields
+            file_fields = (
+                'file',
+                'media_type',
+                'href',
+                'checksum_multihash',
+                'update_interval',
+                'displayed_file_size'
+            )
+
+        return [
+            (None, {
+                'fields': ('name', 'collection_name', 'created', 'updated', 'etag')
+            }),
+            ('File', {
+                'fields': file_fields
+            }),
+            ('Description', {
+                'fields': ('title', 'description', 'roles')
+            }),
+        ]
 
 
 class AssetAdminForm(forms.ModelForm):
@@ -574,7 +670,7 @@ def queryset(self, request, queryset):
 
 
 @admin.register(Asset)
-class AssetAdmin(admin.ModelAdmin):
+class AssetAdmin(AssetUploadAdminMixin, RedirectAfterCreationMixin, admin.ModelAdmin):
     form = AssetAdminForm
 
     class Media:
@@ -753,70 +849,6 @@ def get_form(self, request, obj=None, change=False, **kwargs):
         form = super().get_form(request, obj, change, **kwargs)
         return form
 
-    def response_add(self, request, obj, post_url_continue=None):
-        """
-        Determine the HttpResponse for the add_view stage. It mostly defers to
-        its superclass implementation but is customized because the User model
-        has a slightly different workflow.
-        """
-        # We should allow further modification of the user just added i.e. the
-        # 'Save' button should behave like the 'Save and continue editing'
-        # button except for:
-        # * The user has pressed the 'Save and add another' button
-
-        # stole this from django.contrib.auth.admin.UserAdmin
-        if "_addanother" not in request.POST:
-            request.POST = request.POST.copy()
-            request.POST["_continue"] = 1
-
-        return super().response_add(request, obj, post_url_continue)
-
-    def get_urls(self):
-        urls = super().get_urls()
-        my_urls = [
-            path(
-                "<path:object_id>/change/upload/",
-                self.admin_site.admin_view(self.upload_view),
-                name=f'{self.model._meta.app_label}_{self.model._meta.model_name}_upload',
-            )
-        ]
-        return my_urls + urls
-
-    def upload_view(self, request, object_id, extra_context=None):
-        model = self.model
-        obj = self.get_object(request, unquote(object_id))
-        if obj is None:
-            return self._get_obj_does_not_exist_redirect(request, model._meta, object_id)
-
-        context = dict(
-            # Include common variables for rendering the admin template.
-            self.admin_site.each_context(request),
-            # Anything else you want in the context...
-            csrf_token=request.META['CSRF_COOKIE'],
-            asset_name=obj.name,
-            item_name=obj.item.name,
-            collection_name=obj.get_collection()
-        )
-        return TemplateResponse(request, "uploadtemplate.html", context)
-
-    def change_view(self, request, object_id, form_url='', extra_context=None):
-        obj = Asset.objects.filter(id=request.resolver_match.kwargs['object_id']).first()
-        if obj.is_external:
-            return super().change_view(request, object_id, form_url)
-
-        extra_context = extra_context or {}
-
-        # Generate the transfer URL
-        property_upload_url = reverse(
-            f'admin:{self.model._meta.app_label}_{self.model._meta.model_name}_upload',
-            args=[object_id],
-        )
-
-        # Add the property upload URL to the extra context
-        extra_context['property_upload_url'] = property_upload_url
-
-        return super().change_view(request, object_id, form_url, extra_context=extra_context)
-
 
 @admin.register(AssetUpload)
 class AssetUploadAdmin(admin.ModelAdmin):
diff --git a/app/stac_api/templates/js/admin/upload.js b/app/stac_api/templates/js/admin/upload.js
index fe26881e..8bbacfa2 100644
--- a/app/stac_api/templates/js/admin/upload.js
+++ b/app/stac_api/templates/js/admin/upload.js
@@ -6,7 +6,14 @@ function setStatus(text) {
 }
 
 function getAssetUploadUrlPrefix(collection, item, asset) {
-    return `/api/stac/v0.9/collections/${collection}/items/${item}/assets/${asset}/uploads`
+    const is_collection_asset = (item === '');
+    let item_str = '';
+
+    if (!is_collection_asset) {
+        item_str = `items/${item}/`;
+    }
+
+    return `/api/stac/v0.9/collections/${collection}/${item_str}assets/${asset}/uploads`;
 }
 
 function setError(text) {
diff --git a/app/stac_api/urls.py b/app/stac_api/urls.py
index 5ccaf6cd..36a9ac1a 100644
--- a/app/stac_api/urls.py
+++ b/app/stac_api/urls.py
@@ -87,12 +87,6 @@
     path("<collection_name>/assets/", include(collection_asset_urls))
 ]
 
-collection_urls_v09 = [
-    path("<collection_name>", CollectionDetail.as_view(), name='collection-detail'),
-    path("<collection_name>/items", ItemsList.as_view(), name='items-list'),
-    path("<collection_name>/items/", include(item_urls))
-]
-
 urlpatterns = [
     # Deactivate healthcheck for now while monitoring is being adapted.
     # path(f"{HEALTHCHECK_ENDPOINT}", CollectionList.as_view(), name='health-check'),
@@ -104,7 +98,7 @@
             path("conformance", ConformancePageDetail.as_view(), name='conformance'),
             path("search", SearchList.as_view(), name='search-list'),
             path("collections", CollectionList.as_view(), name='collections-list'),
-            path("collections/", include(collection_urls_v09)),
+            path("collections/", include(collection_urls)),
             path("update-extent", recalculate_extent)
         ],
                  "v0.9"),


Reviews:
user: login: asteiner-swisstopo
body: I must admit that I don't really follow what most of this new code does but then I see that it's copy-paste of the existing asset upload. So looks good to me.

Would be nice if there was less duplication, maybe we can still find a better solution there.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T10:46:42Z
user: login: sami-nouidri-swisstopo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:36:44Z
user: login: adk-swisstopo
body: Once you have addressed the comments, please update the PR description with details of how you verified it works.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T15:22:40Z
user: login: msom
body: The admin UI doesn't allow to create a new collection asset (Error `Invalid media type ""`).
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T13:07:27Z
user: login: msom
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T13:13:43Z
user: login: msom
body: e2e tests (https://github.com/geoadmin/infra-e2e-tests/pull/352) work now :+1: 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T06:06:18Z
user: login: msom
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T06:37:23Z

Review comments:
diff_hunk: @@ -6,6 +6,10 @@ function setStatus(text) {
 }
 
 function getAssetUploadUrlPrefix(collection, item, asset) {
+    if (item === '') {
+        //no item means we're dealing with a collection asset
path: app/stac_api/templates/js/admin/upload.js
user: login: asteiner-swisstopo
body: How about expressing this in a variable name rather than a comment?

Something like

```js
is_collection_asset = item === ''
if (is_collection_asset) {
   ....
}
```
created_at: 2025-03-25T10:19:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -492,6 +503,40 @@ def get_fieldsets(self, request, obj=None):
         fields[0][1]['fields'] = ('name', 'collection_name', 'created', 'updated', 'etag')
         return fields
 
+    def upload_view(self, request, object_id, extra_context=None):
+        model = self.model
+        obj = self.get_object(request, unquote(object_id))
+        if obj is None:
+            return self._get_obj_does_not_exist_redirect(request, model._meta, object_id)
+
+        context = dict(
+            # Include common variables for rendering the admin template.
+            self.admin_site.each_context(request),
+            # Anything else you want in the context...
+            csrf_token=request.META['CSRF_COOKIE'],
+            asset_name=obj.name,
+            collection_name=obj.get_collection()
+        )
+        return TemplateResponse(request, "uploadtemplate.html", context)
+
+    def change_view(self, request, object_id, form_url='', extra_context=None):
+        obj = CollectionAsset.objects.filter(id=request.resolver_match.kwargs['object_id']).first()
+        if obj.is_external:
+            return super().change_view(request, object_id, form_url)
+
+        extra_context = extra_context or {}
+
+        # Generate the transfer URL
+        property_upload_url = reverse(
+            f'admin:{self.model._meta.app_label}_{self.model._meta.model_name}_upload',
+            args=[object_id],
+        )
+
+        # Add the property upload URL to the extra context
+        extra_context['property_upload_url'] = property_upload_url
+
+        return super().change_view(request, object_id, form_url, extra_context=extra_context)
path: app/stac_api/admin.py
user: login: asteiner-swisstopo
body: I see that these two functions, including the `get_urls()` above, are copy-paste of the same functions in `AssetAdmin` (except `Asset` <-> `CollectionAsset`). Have you had a look at how to avoid this duplication?
created_at: 2025-03-25T10:35:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -492,6 +503,40 @@ def get_fieldsets(self, request, obj=None):
         fields[0][1]['fields'] = ('name', 'collection_name', 'created', 'updated', 'etag')
         return fields
 
+    def upload_view(self, request, object_id, extra_context=None):
+        model = self.model
+        obj = self.get_object(request, unquote(object_id))
+        if obj is None:
+            return self._get_obj_does_not_exist_redirect(request, model._meta, object_id)
+
+        context = dict(
+            # Include common variables for rendering the admin template.
+            self.admin_site.each_context(request),
+            # Anything else you want in the context...
+            csrf_token=request.META['CSRF_COOKIE'],
+            asset_name=obj.name,
+            collection_name=obj.get_collection()
+        )
+        return TemplateResponse(request, "uploadtemplate.html", context)
+
+    def change_view(self, request, object_id, form_url='', extra_context=None):
+        obj = CollectionAsset.objects.filter(id=request.resolver_match.kwargs['object_id']).first()
+        if obj.is_external:
+            return super().change_view(request, object_id, form_url)
+
+        extra_context = extra_context or {}
+
+        # Generate the transfer URL
+        property_upload_url = reverse(
+            f'admin:{self.model._meta.app_label}_{self.model._meta.model_name}_upload',
+            args=[object_id],
+        )
+
+        # Add the property upload URL to the extra context
+        extra_context['property_upload_url'] = property_upload_url
+
+        return super().change_view(request, object_id, form_url, extra_context=extra_context)
path: app/stac_api/admin.py
user: login: sami-nouidri-swisstopo
body: I've addressed this issue with a mixin for both
created_at: 2025-03-25T12:36:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -6,6 +6,10 @@ function setStatus(text) {
 }
 
 function getAssetUploadUrlPrefix(collection, item, asset) {
+    is_collection_asset = (item === '');
+    if (is_collection_asset) {
+        return `/api/stac/v0.9/collections/${collection}/assets/${asset}/uploads`
+    }
     return `/api/stac/v0.9/collections/${collection}/items/${item}/assets/${asset}/uploads`
path: app/stac_api/templates/js/admin/upload.js
user: login: adk-swisstopo
body: I deny any knowledge about JavaScript but I think a single return path is generally cleaner. E.g.
```
item_str = ''
if (is_collection_asset) {
  item_str = 'items/${item}/'
}
return '/api/stac/v0.9/collections/${collections}/${item_str}assets/${asset}/uploads'
```
created_at: 2025-03-25T14:33:32Z
author_association: MEMBER
diff_hunk: @@ -221,3 +225,54 @@ def patch_collections_aggregate_cache_control_header(response):
     else:
         patch_response_headers(response, cache_timeout=settings.COLLECTIONS_AGGREGATE_CACHE_SECONDS)
         patch_cache_control(response, public=True)
+
+
+class AssetUploadAdminMixin:
path: app/stac_api/views/mixins.py
user: login: adk-swisstopo
body: That doesn't look like a view. Why is it in views?
If it's only used within admin.py I think it's fine to keep it in that file.
created_at: 2025-03-25T15:21:27Z
author_association: MEMBER
diff_hunk: @@ -491,21 +542,45 @@ def displayed_file_size(self, instance):
     # That's why some fields like the name of the asset are set readonly here
     # for update operations
     def get_fieldsets(self, request, obj=None):
-        fields = super().get_fieldsets(request, obj)
+        """Build the different field sets for the admin page."""
+
+        base_fields = super().get_fieldsets(request, obj)
+
         if obj is None:
-            # In case a new Asset is added use the normal field 'collection' from model that have
-            # a help text fort the search functionality.
-            fields[0][1]['fields'] = ('name', 'collection', 'created', 'updated', 'etag')
-            return fields
-        # Otherwise if this is an update operation only display the read only fields
-        # without help text
+            return ((None, {'fields': ('name', 'collection', 'created', 'updated', 'etag')}),)
+
path: app/stac_api/admin.py
user: login: msom
body: `media_type` is missing, see https://github.com/geoadmin/service-stac/blob/develop/app/stac_api/admin.py#L703
created_at: 2025-03-27T13:13:42Z
author_association: CONTRIBUTOR

PR comments:
user: login: msom
created_at: 2025-03-27T13:21:17Z
body: Creating a new collection asset should probably redirect to the edit form (2-step), similar to assets: https://github.com/geoadmin/service-stac/blob/develop/app/stac_api/admin.py#L756
author_association: CONTRIBUTOR
user: login: sami-nouidri-swisstopo
created_at: 2025-03-28T14:05:13Z
body: > The admin UI doesn't allow to create a new collection asset (Error `Invalid media type ""`).

I've made the changes requested, and tested the big asset upload feature, which seems to work correctly. 

I've however encountered a peculiar case of flakyness, where **sometimes** the end-to-end tests failed when using the `debug` docker image on the k8s cluster. I could not replicate this behavior locally, and the tests pass when using the docker `prod` image.

Here's the error that would sometimes occur : 
`AssertionError: 500 not found in [200, 404] : ClientError('An error occurred (AccessDenied) when calling the DeleteObject operation: User: arn:aws:iam::839910802816:user/service-stac-tech-dev is not authorized to perform: s3:DeleteObject on resource: "arn:aws:s3:::service-stac-managed-dev-swisstopo/test-e2e-asset-upload-f2c004cd-71cc-48cb-a8f2-aadbfc5b7fd3/test-e2e-asset-upload-64857f86-108f-469f-9996-ba9c5d4d3412/test-e2e-asset-upload-0db0740d-a7d3-4fa0-8785-28454d62ae00.txt" because no identity-based policy allows the s3:DeleteObject action')`
author_association: CONTRIBUTOR
user: login: adk-swisstopo
created_at: 2025-03-31T05:49:04Z
body: > **sometimes** the end-to-end tests failed when using the `debug` docker image on the k8s cluster

Sounds like PB-1249. AFAICT the debug image was never meant to pass the end-to-end test.
author_association: MEMBER