Package: indico-plugin-storage-s3
S3 Storage PluginThe S3 storage plugin allows Indico to store materials etc. onAmazon S3or
an S3-compatible object storage service (Ceph S3, Minio, etc.) instead of the local file
system.WarningThis plugin has only been tested with Ceph S3 so far. So if you encounter any problems
using e.g. the real Amazon S3, please let us know!It is currently used in production on multiple Indico instances, so we believe it is
stable, but please be advised that we do not provide a way to move files back from S3
to local storage (but it would of course be possible to write a script for this).Changelog3.2.2Support Python 3.113.2.1Stop using deprecated URL utils from werkzeug3.2Update translations3.1.2No technical changes, just fixing a mistake in the README change from 3.1.13.1.1No technical changes, just adding the missing README to PyPI and updating the nginx
config snippet to correctly work with the changes from 3.1 (avoiding annginx bug)3.1Fix "invalid signature" S3 error in some cases when usingproxy=nginxfor downloads3.0Initial release for Indico 3.0ConfigurationConfiguration is done using theSTORAGE_BACKENDSentry ofindico.conf; add a new key
with a name of your choice (e.g.s3) and specify the details of the S3 storage in the
value.For a single bucket, all you need to specify is the bucket name:STORAGE_BACKENDS={# ...'s3':'s3:bucket=indico-test'}If you want to dynamically create buckets for each year, month or week, you can do this
as well. A task will automatically create new buckets a while before it will become active.STORAGE_BACKENDS={# ...'s3':'s3-dynamic:bucket_template=indico-test-<year>,bucket_secret=somethingrandom'}For authentication and general S3 config (e.g. to use subdomains for bucket names), the
preferred way is to use the standard files, i.e.~/.aws/credentialsand~/.aws/config,
but you can also specify all settings in the storage backend entry like this:STORAGE_BACKENDS={# ...'s3':'s3:bucket=my-indico-test-bucket,access_key=12345,secret_key=topsecret'}Available config optionsMultiple options can be specified by separating them with commas. These options are available:host-- the host where S3 is running, in case you use a custom S3-compatible storage.profile-- the name of a specific S3 profile (used in the~/.aws/config files)access_key-- the S3 access key; should not be used in favor of~/.aws/credentialssecret_key-- the S3 access key; should not be used in favor of~/.aws/credentialsaddressing_style-- the S3 addressing style (virtualorpath); should not be used in
favor of~/.aws/configbucket_policy_file-- the path to a file containing an S3 bucket policy; this only applies
to new buckets created by this pluginbucket_versioning-- whether to enable S3 versioning on the bucket; this only applies to new
buckets created by this pluginproxy-- whether to proxy downloads. If set totrue, all files will be downloaded to memory
and then sent to the client by Indico. This may have performance implications if you have large
files. A better option is setting it tonginx, which requires some extra configuration (see below),
but lets nginx handle proxying downloads transparently. If you do not use proxying at all,
downloading a file redirects the user to a temporary S3 URL valid for a few minutes. Generally
this works fine, but it may result in people accidentally copying (and forwarding) temporary
links that expire quickly.meta-- a custom string that is included in the bucket info API of the plugin. You generally do
not need this unless you are using custom scripts accessing that API and want to include some
extra data there.When using thes3backend (single static bucket), the following extra option is available:bucket(required) -- the name of the bucketWhen using thes3-dynamicbackend, the following extra options are available:bucket_template(required) -- a template specifying how the bucket names should be generated.
Needs to contain at least one of<year>,<month>or<week>bucket_secret(required unless set in aws config) -- a random secret used to make bucket names
unguessable (as bucket names need to be globally unique on S3); may also be specified asindico_bucket_secretin~/.aws/credentialsProxying downloads through nginxIf you want to use theproxy=nginxoption to avoid redirecting users to the actual S3 URL
for file downloads without having the extra load and memory usage of downloading a (possibly
large) attachment to memory first that comes withproxy=on, you need to add the following
to theserverblock in your nginx config that is responsible for Indico.location~^/\.xsf/s3/(?<download_protocol>https?)/(?<download_host>[^/]+)/(?<download_path>.+)${internal;set$download_url$download_protocol://$download_host/$download_path;resolverYOUR_RESOLVER;proxy_set_headerHost$download_host;proxy_set_headerAuthorization'';proxy_set_headerCookie'';proxy_hide_headerX-Amz-Request-Id;proxy_hide_headerBucket;proxy_max_temp_file_size0;proxy_intercept_errorson;error_page301302307=@s3_redirect;proxy_pass$download_url$is_args$args;}location@s3_redirect{internal;resolverYOUR_RESOLVER;set$saved_redirect_location'$upstream_http_location';proxy_pass$saved_redirect_location;}ReplaceYOUR_RESOLVERwith the hostname or IP address of a nameserver nginx can use to
resolve the S3 hosts. You may find a suitable IP in your/etc/resolv.confor by asking
someone from your IT department. If you are running a local caching nameserver,localhostwould work as well.If you are interested in how this works, checkthis blog poston which this
config is based.Migration of existing dataThe plugin comes with a migration tool, accessible through theindico s3 migrateCLI.
It can be used without downtime of your service as it consists of two steps - first copying
the files, and then updating references in your database. Please have a look at its--helpoutput if you want to use it; we did not have time to write detailed documentation for it yet.The step that updates the database can be reversed in case you want to switch back from S3
to local storage for whatever reason, but it will only affect migrated files - any file stored
directly on S3 later (and thus not present on the local file system), will not be reverted.
You would need to write your own script that downloads those files from S3.