Project: Alluxio (Runtime / Cloud Native Storage)
File: S3.md

layout: global title: Amazon AWS S3 This guide describes the instructions to configure as Alluxio's under storage system. Amazon AWS S3, or Amazon Simple Storage Service, is an object storage service offering industry-leading scalability, data availability, security, and performance. For more information about Amazon AWS S3, please read its {:target="_blank"}. If you haven't already, please see before you get started. In preparation for using Amazon AWS S3 with Alluxio: <table class="table table-striped"> <tr> <td markdown="span" style="width:30%">`<S3_BUCKET>`</td> <td markdown="span">{:target="_blank"} or use an existing bucket</td> </tr> <tr> <td markdown="span" style="width:30%">`<S3_DIRECTORY>`</td> <td markdown="span">The directory you want to use in that container, either by creating a new directory or using an existing one. </td> </tr> <tr> <td markdown="span" style="width:30%">`<S3ACCESSKEY_ID>`</td> <td markdown="span">Used to sign programmatic requests made to AWS. See {:target="_blank"}</td> </tr> <tr> <td markdown="span" style="width:30%">`<S3SECRETKEY>`</td> <td markdown="span">Used to sign programmatic requests made to AWS. See {:target="_blank"}</td> </tr> </table> To use Amazon AWS S3 as the UFS of Alluxio root mount point, you need to configure Alluxio to use under storage systems by modifying `conf/alluxio-site.properties`. If it does not exist, create the configuration file from the template. ```shell $ cp conf/alluxio-site.properties.template conf/alluxio-site.properties ``` Specify an existing S3 bucket and directory as the underfs address by modifying `conf/alluxio-site.properties` to include: ```properties alluxio.dora.client.ufs.root=s3://<S3BUCKET>/<S3DIRECTORY> ``` Note that if you want to mount the whole s3 bucket, add a trailing slash after the bucket name (e.g. `s3://S3_BUCKET/`). Specify the AWS credentials for S3 access by setting `s3a.accessKeyId` and `s3a.secretKey` in `alluxio-site.properties`. ```properties s3a.accessKeyId=<S3ACCESSKEY_ID> s3a.secretKey=<S3SECRETKEY> ``` For other methods of setting AWS credentials, see the credentials section in . Once you have configured Alluxio to Amazon AWS S3, try to see that everything works. Configure S3 region when accessing S3 buckets to improve performance. Otherwise, global S3 bucket access will be enabled which introduces extra requests. S3 region can be set in `conf/alluxio-site.properties` ```properties alluxio.underfs.s3.region=us-west-1 ``` You can specify credentials in different ways, from highest to lowest priority: `s3a.accessKeyId` and `s3a.secretKey` specified as mount options `s3a.accessKeyId` and `s3a.secretKey` specified as Java system properties `s3a.accessKeyId` and `s3a.secretKey` in `alluxio-site.properties` Environment Variables `AWSACCESSKEYID` or `AWSACCESS_KEY` (either is acceptable) and `AWSSECRETACCESSKEY` or `AWSSECRET_KEY` (either is acceptable) on the Alluxio servers Profile file containing credentials at `~/.aws/credentials` AWS Instance profile credentials, if you are using an EC2 instance When using an AWS Instance profile as the credentials' provider: Create an with access to the mounted bucket Create an as a container for the defined IAM Role Launch an EC2 instance using the created profile Note that the IAM role will need access to both the files in the bucket as well as the bucket itself in order to determine the bucket's owner. Automatically assigning an owner to the bucket can be avoided by setting the property `alluxio.underfs.s3.inherit.acl=false`. See for more details. You may encrypt your data stored in S3. The encryption is only valid for data at rest in S3 and will be transferred in decrypted form when read by clients. Note, enabling this will also enable HTTPS to comply with requirements for reading/writing objects. Enable this feature by configuring `conf/alluxio-site.properties`: ```properties

``` By default, a request directed at the bucket named "mybucket" will be sent to the host name "mybucket.s3.amazonaws.com". You can enable DNS-Buckets to use path style data access, for example: "http://s3.amazonaws.com/mybucket" by setting the following configuration: ```properties alluxio.underfs.s3.disable.dns.buckets=true ``` To communicate with S3 through a proxy, modify `conf/alluxio-site.properties` to include: ```properties alluxio.underfs.s3.proxy.host=<PROXY_HOST> alluxio.underfs.s3.proxy.port=<PROXY_PORT> ``` `<PROXYHOST>` and `<PROXYPORT>` should be replaced by the host and port of your proxy. To use an S3 service provider other than "s3.amazonaws.com", modify `conf/alluxio-site.properties` to include: ```properties alluxio.underfs.s3.endpoint=<S3_ENDPOINT> alluxio.underfs.s3.endpoint.region=<S3ENDPOINTREGION> ``` Replace `<S3_ENDPOINT>` with the hostname and port of your S3 service, e.g., `http://localhost:9000`. Only use this parameter if you are using a provider other than `s3.amazonaws.com`. Both the endpoint and region value need to be updated to use non-home region. ```properties alluxio.underfs.s3.endpoint=<S3_ENDPOINT> alluxio.underfs.s3.endpoint.region=<S3ENDPOINTREGION> ``` All OCI object storage regions need to use `PathStyleAccess` ```properties alluxio.underfs.s3.disable.dns.buckets=true alluxio.underfs.s3.inherit.acl=false ``` Some S3 service providers only support v2 signatures. For these S3 providers, you can enforce using the v2 signatures by setting the `alluxio.underfs.s3.signer.algorithm` to `S3SignerType`. S3 is an object store and because of this feature, the whole file is sent from client to worker, stored in the local disk temporary directory, and uploaded in the `close()` method by default. To enable S3 streaming upload, you need to modify `conf/alluxio-site.properties` to include: ```properties alluxio.underfs.s3.streaming.upload.enabled=true ``` The default upload process is safer but has the following issues: Slow upload time. The file has to be sent to Alluxio worker first and then Alluxio worker is responsible for uploading the file to S3. The two processes are sequential. The temporary directory must have the capacity to store the whole file. Slow `close()`. The execution time of `close()` method is proportional to the file size and inversely proportional to the bandwidth. That is O(FILE_SIZE/BANDWIDTH). Slow `close()` is unexpected and has already been a bottleneck in the Alluxio Fuse integration. Alluxio Fuse method which calls `close()` is asynchronous and thus if we write a big file through Alluxio Fuse to S3, the Fuse write operation will be returned much earlier than the file has been written to S3. The S3 streaming upload feature addresses the above issues and is based on the . The S3 streaming upload has the following advantages: Shorter upload time. Alluxio worker uploads buffered data while receiving new data. The total upload time will be at least as fast as the default method. Smaller capacity requirement. Our data is buffered and uploaded according to partitions (`alluxio.underfs.s3.streaming.upload.partition.size` which is 64MB by default). When a partition is successfully uploaded, this partition will be deleted. Faster `close()`. We begin uploading data when data buffered reaches the partition size instead of uploading the whole file in `close()`. If a S3 streaming upload is interrupted, there may be intermediate partitions uploaded to S3 and S3 will charge for those data. To reduce the charges, users can modify `conf/alluxio-site.properties` to include: ```properties alluxio.underfs.cleanup.enabled=true ``` Intermediate multipart uploads in all non-readonly S3 mount points older than the clean age (configured by `alluxio.underfs.s3.intermediate.upload.clean.age`) will be cleaned when a leading master starts or a cleanup interval (configured by `alluxio.underfs.cleanup.interval`) is reached. The default upload method uploads one file completely from start to end in one

We use multipart-upload method to upload one file by multiple parts, every part will be uploaded in one thread. It won't generate any temporary files while uploading. It will consume more memory but faster than streaming upload mode. To enable S3 multipart upload, you need to modify `conf/alluxio-site.properties` to include: ```properties alluxio.underfs.s3.multipart.upload.enabled=true ``` There are other parameters you can specify in `conf/alluxio-site.properties` to make the process faster and better. ```properties alluxio.underfs.object.store.multipart.upload.timeout ``` ```properties alluxio.underfs.s3.multipart.upload.partition.size ``` When using Alluxio to access S3 with a great number of clients per Alluxio server, these parameters can be tuned so that Alluxio uses a configuration optimized for the S3 backend. If the S3 connection is slow, a larger timeout is useful: ```properties alluxio.underfs.s3.socket.timeout=500sec alluxio.underfs.s3.request.timeout=5min ``` If we expect a great number of concurrent metadata operations: ```properties alluxio.underfs.s3.admin.threads.max=80 ``` If the total number of metadata + data operations is huge: ```properties alluxio.underfs.s3.threads.max=160 ``` For a worker, the number of concurrent writes to S3. For a master, the number of threads to concurrently rename files within a directory. ```properties alluxio.underfs.s3.upload.threads.max=80 ``` Thread-pool size to submit delete and rename operations to S3 on master: ```properties alluxio.underfs.object.store.service.threads=80 ``` is very different from the traditional POSIX permission model. For instance, S3 ACL does not support groups or directory-level settings. Alluxio makes the best effort to inherit permission information including file owner, group and permission mode from S3 ACL information. The S3 credentials set in Alluxio configuration corresponds to an AWS user. If this user does not have the required permissions to access an S3 bucket or object, a 403 permission denied error will be returned. If you see a 403 error in Alluxio server log when accessing an S3 service, you should double-check You are using the correct AWS credentials. See . Your AWS user has permissions to access the buckets and objects mounted to Alluxio. Read more for 403 error. Alluxio file system sets the file owner based on the AWS account configured in Alluxio to connect to S3. Since there is no group in S3 ACL, the owner is reused as the group. By default, Alluxio extracts the display name of this AWS account as the file owner. In case this display name is not available, this AWS user's will be used. This canonical user ID is typically a long string (like `79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be`), thus often inconvenient to read and use in practice. Optionally, the property `alluxio.underfs.s3.owner.id.to.username.mapping` can be used to specify a preset mapping from canonical user IDs to Alluxio usernames, in the format "id1=user1;id2=user2". For example, edit `alluxio-site.properties` to include ```properties alluxio.underfs.s3.owner.id.to.username.mapping=\ 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be=john ``` This configuration helps Alluxio recognize all objects owned by this AWS account as owned by the user `john` in Alluxio namespace. To find out the AWS S3 canonical ID of your account, check the console `https://console.aws.amazon.com/iam/home?#/security_credentials`, expand the "Account Identifiers" tab and refer to "Canonical User ID". `chown`, `chgrp`, and `chmod` of Alluxio directories and files do NOT propagate to the underlying S3 buckets nor objects. If issues are encountered when running against your S3 backend, enable additional logging to track HTTP traffic. Modify `conf/log4j.properties` to add the following properties: ```properties log4j.logger.com.amazonaws=WARN log4j.logger.com.amazonaws.request=DEBUG log4j.logger.org.apache.http.wire=DEBUG ``` See for more details.