Package: oarepo-s3
oarepo-s3This package built on top of theinvenio-s3library offers integration with any AWS S3 REST API compatible object storage backend.
In addition to the invenio-s3, it tries to minimize processing of file requests on the
Invenio server side and uses direct access to S3 storage backend as much as possible
(neither multipart file uploads, nor downloads are processed by Invenio server itself).InstalationTo start using this libraryinstall the following packages in your project's venv:gitclonehttps://github.com/CESNET/s3-clientcds3-client
poetryinstall
pipinstalloarepo-s3Create an S3 account and bucket on your S3 storage provider of choice.Put the S3 access configuration into your Invenio server config (e.g.invenio.cfg):INVENIO_S3_TENANT=NoneINVENIO_S3_ENDPOINT_URL='https://s3.example.org'INVENIO_S3_ACCESS_KEY_ID='your_access_key'INVENIO_S3_SECRET_ACCESS_KEY='your_secret_key'Create Invenio files location targetting the S3 bucketinveniofileslocation--default'default-s3's3://oarepo-bucketUsageTo use this library as an Invenio Files storage in your projects, put the following
into your Invenio server config:FILES_REST_STORAGE_FACTORY='oarepo_s3.storage.s3_storage_factory'This storage overrides thesave()method from theInvenioS3storage and adds
the possibility fordirect S3 multipart uploads. Every other functionality
is handled by underlyingInvenioS3storage library.Direct multipart uploadTo create a direct multipart upload to S3 backend, one should provide an
instance ofMultipartUploadinstead of a usualstreamwhen assigning
a file to a certain record, e.g.:fromoarepo_s3.apiimportMultipartUploadfiles=record.files# Record instance FilesIteratormu=MultipartUpload(key='filename',base_uri=files.bucket.location.uri,expires=3600,size=1024*1024*1000)# total file size# Assigning a MultipartUpload to the FilesIterator here will# trigger the multipart upload creation on the S3 storage backend.files['test']=muthis will configure the passed inMultipartUploadinstance with
all the information needed by any uploader client to process and
complete the upload. The multipart upload session configuration
can be found under theMultipartUpload.sessionfield.To be able to complete or abort an ongoing multipart upload, after an
uploader client finishes uploading all the parts to the S3 backend,
one needs to register the provided resources fromoarepo_s3.viewsin
the app blueprints:Create multipart uploadfiles/<key>/?multipart=TrueCreate presigned URL for part uploadfiles/<key>/<upload_id>/<part_num>/presignedList uploaded parts for a given multipart uploadfiles/<key>/<upload_id>/partsComplete a multipart uploadfiles/<key>/<upload_id>/completeAbort an ongoing multipart uploadfiles/<key>/<upload_id>/abortOARepo Records Draft integrationThis library works best together withoarepo-records-draftlibrary. When integrated into draft endpoints one doesn't need to manually
register the completion resources to blueprints. Multipart upload creation
is also handled automatically.To setup a drafts integration, just run the following:pipinstalloarepo-records-draftoarepo-s3and configure draft endpoints according to the library's README.
Doing so, will auto-register the following file API actions on the draft
endpoints:Create multipart uploadPOST /draft/records/<pid>/files/?multipart=True{"key": "filename.txt","ctype": "text/plain","size": 1024}Get presigned URL for part uploadGET /draft/records/<pid>/files/<key>/<upload_id>/<part_num>/presignedList uploaded parts for a given multipart uploadGET /draft/records/<pid>/files/<key>/<upload_id>/partsComplete multipart uploadPOST /draft/records/<pid>/files/<key>/<upload_id>/complete{"parts": [{"ETag": <uploaded_part_etag>, "PartNumber": <part_num>},...]}Abort multipart uploadDELETE /draft/records/<pid>/files/<key>/<upload_id>/abortTasksThis library provides a task that looks up the expired ongoing
file uploads that could no longer be completed and removes them
from the associated record's bucket, to use this task in your
Celery cron schedule, configure it in your Invenio server config like this:CELERY_BEAT_SCHEDULE={'cleanup_expired_multipart_uploads':{'task':'oarepo_s3.tasks.cleanup_expired_multipart_uploads','schedule':timedelta(minutes=60),},...}