Project: Alluxio (Runtime / Cloud Native Storage)
File: S3-API.md

layout: global title: S3 API Alluxio supports a that is compatible with the basic operations of the Amazon {:target="_blank"}. The Alluxio S3 API should be used by applications designed to communicate with an S3-like storage and would benefit from the other features provided by Alluxio, such as data caching, data sharing with file system based applications, and storage system abstraction (e.g., using Ceph instead of S3 as the backing store). For example, a simple application that downloads reports generated by analytic tasks can use the S3 API instead of the more complex file system API. Only top-level Alluxio directories are treated as buckets by the S3 API. Hence the root directory of the Alluxio filesystem is not treated as an S3 bucket. Any root-level objects (eg: `alluxio://file`) will be inaccessible through the Alluxio S3 API. To treat sub-directories as a bucket, the separator `:` must be used in the bucket name (eg: `s3://sub:directory:bucket/file`). Note that this is purely a convenience feature and hence is not returned by API Actions such as ListBuckets. Alluxio uses `/` as a reserved separator. Therefore, any S3 paths with objects or folders named `/` (eg: `s3://example-bucket//`) will cause undefined behavior. For additional limitations on object key names please check this page: {:target="_blank"} is not supported in the Alluxio S3 API. Therefore, S3 clients must utilize {:target="_blank"} (i.e: `http://s3.amazonaws.com/{bucket}/{object}` and NOT `http://{bucket}.s3.amazonaws.com/{object}`). As described in the AWS S3 docs for {:target="_blank"}: Amazon S3 is a distributed system. If it receives multiple write requests for the same object simultaneously, it overwrites all but the last object written. Amazon S3 does not provide object locking; if you need this, make sure to build it into your application layer or use versioning instead. Note that at the moment the Alluxio S3 API does not support object versioning Alluxio S3 will overwrite the existing key and the temporary directory for multipart upload. All sub-directories in Alluxio will be returned in ListObjects(V2) as 0-byte folders. This behavior is in accordance with if you used the AWS S3 console to create all parent folders for each object. User-defined tags on buckets & objects are limited to 10 and obey the {:target="_blank"}. Set the property key `alluxio.proxy.s3.tagging.restrictions.enabled=false` to disable this behavior. The maximum size for user-defined metadata in PUT-requests is 2KB by default in accordance with {:target="_blank"}. Set the property key `alluxio.proxy.s3.header.metadata.max.size` to change this behavior. The S3 API leverages the , introducing an additional network hop for Alluxio clients. For optimal performance, it is recommended to run the proxy server and an Alluxio worker on each compute node. It is also recommended to put all the proxy servers behind a load balancer. <table class="table table-striped"> <tr> <th>Header</th> <th>Content</th> <th>Description</th> </tr> <tr> <td><a href="https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html" target="_blank">Authorization</a></td> <td>AWS4-HMAC-SHA256 Credential=<b>{user}</b>/..., SignedHeaders=..., Signature=...</td> <td>There is currently no support for access & secret keys in the Alluxio S3 API. The only supported authentication scheme is the SIMPLE authentication type. By default, the user that is used to perform any operations is the user that was used to launch the Alluxio proxy process. <br/><br/> Therefore this header is used exclusively to specify an Alluxio ACL username to perform an operation with. In order to remain compatible with other S3 clients, the header is still expected to follow the <a

target="_blank">AWS Signature Version 4</a> format. <br/><br/> When supplying an access key to an S3 client, put the intended Alluxio ACL username. The secret key is unused so you may use any dummy value.</td> </tr> </table> The following table describes the support status for current {:target="_blank"}: <table class="table table-striped"> <tr> <th>S3 API Action</th> <th>Supported Headers</th> <th>Supported Query Parameters</th> </tr> {% for item in site.data.table.s3-api-supported-actions %} <tr> <td><a href="https://docs.aws.amazon.com/AmazonS3/latest/API/API{{ item.action }}.html" target="blank">{{ item.action }}</a></td> <td> {% assign headers = item.headers | split: "|" %} {% if headers.size == 0 %} N/A {% else %} <ul style="list-style-type:none;margin:0;padding:0"> {% for header in headers %} {% if forloop.last %} <li>{{ header }}</li> {% else %} <li>{{ header }},</li> {% endif %} {% endfor %} </ul> {% endif %} </td> <td> {% assign params = item.queryParams | split: "|" %} {% if params.size == 0 %} N/A {% else %} <ul style="list-style-type:none;margin:0;padding:0"> {% for param in params %} {% if forloop.last %} <li>{{ param }}</li> {% else %} <li>{{ param }},</li> {% endif %} {% endfor %} </ul> {% endif %} </td> </tr> {% endfor %} </table> The following table contains the configurable which pertain to the Alluxio S3 API. <table class="table table-striped"> <tr><th>Property Name</th><th>Default</th><th>Description</th></tr> {% for item in site.data.table.common-configuration %} {% if item.propertyName contains "alluxio.proxy.s3" %} <tr> <td><a class="anchor" name="{{ item.propertyName }}"></a> {{ item.propertyName }}</td> <td>{{ item.defaultValue }}</td> <td>{{ site.data.table.en.common-configuration[item.propertyName] }}</td> </tr> {% endif %} {% endfor %} </table> {% navtabs s3apiactions %} {% navtab AWS CLI %} You can use the {:target="_blank"} to send S3 API requests to the Alluxio S3 API. Note that you will have to provide the `--endpoint` parameter to specify the location of the Alluxio S3 REST API with the server's base URI included (i.e: `--endpoint "http://{alluxio.proxy.web.hostname}:{alluxio.proxy.web.port}/api/v1/s3/"`). As a pre-requisite for operations which involve the `Authorization` header you may need to {:target="_blank"}. See the for details on how Alluxio uses this header ```shell $ aws configure --profile alluxio-s3 AWS Access Key ID [None]: {user} AWS Secret Access Key [None]: {dummy value} Default region name [None]: Default output format [None]: ``` {% endnavtab %} {% navtab REST Clients %} You can directly use any HTTP client to send S3 API requests to the Alluxio S3 API. Note that the base URI for the Alluxio S3 API's REST server is `/api/v1/s3/` (i.e: your requests should be directed to `"http://{alluxio.proxy.web.hostname}:{alluxio.proxy.web.port}/api/v1/s3/"`). At the moment, access key and secret key validation does not exist for the Alluxio S3 API. Therefore the is used purely to specify the intended user to perform a request. The header follows the {:target="_blank"} format. ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." ... ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs abortmultipartupload %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-objects-v2 \ --bucket=testbucket { "Contents": [ { "Key": "multipartcopy.txt6367cf96-ea4e-4447-b931-c5bc91200375/", "LastModified": "2022-05-03T13:00:13.429000+00:00", "Size": 0 }, { "Key": "multipartcopy.txt6367cf96-ea4e-4447-b931-c5bc91200375/1", "LastModified": "2022-05-03T13:00:13.584000+00:00", "Size": 27040 }, { "Key": "test.txt", "LastModified": "2022-05-03T11:55:01.925000+00:00", "Size": 27040 } ] } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api abort-multipart-upload \ --bucket=testbucket --key=multipart_copy.txt --upload-id=6367cf96-ea4e-4447-b931-c5bc91200375 $ % aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-objects-v2 \ --bucket=testbucket { "Contents": [ { "Key": "test.txt", "LastModified":

"Size": 27040 } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:45:17 GMT Content-Type: application/xml Content-Length: 583 Server: Jetty(9.4.43.v20210629) <ListBucketResult> <version2>false</version2> <Marker/> <Prefix/> <IsTruncated>false</IsTruncated> <Name>testbucket</Name> <Contents> <Key>multipart.txt_6367cf96-ea4e-4447-b931-c5bc91200375/</Key> <Size>0</Size> <LastModified>2022-05-03T16:44:17.490Z</LastModified> </Contents> <Contents> <Key>multipart.txt_6367cf96-ea4e-4447-b931-c5bc91200375/1</Key> <Size>27040</Size> <LastModified>2022-05-03T16:44:17.715Z</LastModified> </Contents> <Contents> <Key>test.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:47:36.600Z</LastModified> </Contents> <MaxKeys>1000</MaxKeys> <EncodingType>url</EncodingType> </ListBucketResult> $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X DELETE "http://localhost:39999/api/v1/s3/testbucket/multipart.txt?uploadId=6367cf96-ea4e-4447-b931-c5bc91200375" HTTP/1.1 204 No Content Date: Tue, 03 May 2022 23:45:30 GMT Server: Jetty(9.4.43.v20210629) $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:45:36 GMT Content-Type: application/xml Content-Length: 318 Server: Jetty(9.4.43.v20210629) <ListBucketResult> <version2>false</version2> <Marker/> <Prefix/> <IsTruncated>false</IsTruncated> <Name>testbucket</Name> <Contents> <Key>test.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:47:36.600Z</LastModified> </Contents> <MaxKeys>1000</MaxKeys> <EncodingType>url</EncodingType> </ListBucketResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs completemultipartupload %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api complete-multipart-upload \ --bucket=testbucket --key=multipart.txt --upload-id=6367cf96-ea4e-4447-b931-c5bc91200375 \ --multipart-upload="Parts=[{PartNumber=1},{PartNumber=2}]" { "Location": "/testbucket/multipart.txt", "Bucket": "testbucket", "Key": "multipart.txt", "ETag": "911df44b7ff57801ca8d74568e4ebfbe" } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api head-object \ --bucket=testbucket --key=multipart.txt { "LastModified": "2022-05-03T20:01:43+00:00", "ContentLength": 27040, "ETag": "0cc175b9c0f1b6a831c399e269772661", "ContentType": "application/octet-stream", "Metadata": {} } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ cat complete_upload.xml <CompleteMultipartUpload xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Part> <PartNumber>1</PartNumber> </Part> <Part> <PartNumber>2</PartNumber> </Part> </CompleteMultipartUpload> $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -H "Content-Type: application/xml" -d "@complete_upload.xml" \ -X POST "http://localhost:39999/api/v1/s3/testbucket/multipart.txt?uploadId=6367cf96-ea4e-4447-b931-c5bc91200375" Date: Tue, 03 May 2022 23:59:17 GMT Content-Type: text/xml;charset=utf-8 Transfer-Encoding: chunked Server: Jetty(9.4.43.v20210629) <CompleteMultipartUploadResult> <Location>/testbucket/multipart.txt</Location> <Bucket>testbucket</Bucket> <Key>multipart.txt</Key> <ETag>911df44b7ff57801ca8d74568e4ebfbe</ETag> <Code/> <Message/> </CompleteMultipartUploadResult> $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ --head "http://localhost:39999/api/v1/s3/testbucket/multipart.txt" HTTP/1.1 200 OK Date: Wed, 04 May 2022 00:00:40 GMT Last-Modified: Tue, 03 May 2022 23:59:18 GMT ETag: 0cc175b9c0f1b6a831c399e269772661 Content-Type: application/octet-stream Content-Length: 27040 Server: Jetty(9.4.43.v20210629) ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs copy_object %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api copy-object \ --copy-source=testbucket/test.txt --bucket=testbucket --key=test_copy.txt { "CopyObjectResult": { "ETag": "911df44b7ff57801ca8d74568e4ebfbe", "LastModified": "2022-05-03T11:37:16.015000+00:00" } } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-objects \ --bucket=testbucket { "Contents": [ { "Key": "test.txt", "LastModified": "2022-05-03T11:35:59.243000+00:00", "Size": 27040 }, { "Key": "test_copy.txt", "LastModified": "2022-05-03T11:37:16.185000+00:00", "Size": 27040 } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -H "x-amz-copy-source: testbucket/test.txt" \ -X PUT http://localhost:39999/api/v1/s3/testbucket/test_copy.txt HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:50:07 GMT Content-Type: application/xml Content-Length: 135 Server: Jetty(9.4.43.v20210629) <CopyObjectResult> <ETag>911df44b7ff57801ca8d74568e4ebfbe</ETag> <LastModified>2022-05-03T14:50:07.781Z</LastModified> </CopyObjectResult> $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET http://localhost:39999/api/v1/s3/testbucket HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:50:26 GMT Content-Type: application/xml Content-Length: 434 Server: Jetty(9.4.43.v20210629) <ListBucketResult> <version2>false</version2> <Contents> <Key>test.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:47:36.600Z</LastModified> </Contents> <Contents> <Key>test_copy.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:50:07.790Z</LastModified> </Contents> <Marker/> <IsTruncated>false</IsTruncated> <Prefix/> <Name>testbucket</Name> <MaxKeys>1000</MaxKeys> <EncodingType>url</EncodingType> </ListBucketResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs create_bucket %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api create-bucket \ --bucket=testbucket $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-buckets { "Buckets": [ { "Name": "testbucket", "CreationDate": "2022-05-03T11:32:34.156000+00:00" } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X PUT http://localhost:39999/api/v1/s3/testbucket HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:35:05 GMT Content-Length: 0 Server:

$ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET http://localhost:39999/api/v1/s3/ HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:35:23 GMT Content-Type: application/xml Content-Length: 161 Server: Jetty(9.4.43.v20210629) <ListAllMyBucketsResult> <Buckets> <Bucket> <Name>testbucket</Name> <CreationDate>2022-05-03T14:34:56.420Z</CreationDate> </Bucket> </Buckets> </ListAllMyBucketsResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs createmultipartupload %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api create-multipart-upload \ --bucket=testbucket --key=multipart.txt { "Bucket": "testbucket", "Key": "multipart.txt", "UploadId": "6367cf96-ea4e-4447-b931-c5bc91200375" } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X POST "http://localhost:39999/api/v1/s3/testbucket/multipart.txt?uploads" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:41:26 GMT Content-Type: application/xml Content-Length: 147 Server: Jetty(9.4.43.v20210629) <InitiateMultipartUploadResult> <Bucket>testbucket</Bucket> <Key>multipart.txt</Key> <UploadId>6367cf96-ea4e-4447-b931-c5bc91200375</UploadId> </InitiateMultipartUploadResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs delete_bucket %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-buckets { "Buckets": [ { "Name": "tempbucket", "CreationDate": "2022-05-03T11:55:58.134000+00:00" }, { "Name": "testbucket", "CreationDate": "2022-05-03T11:32:34.156000+00:00" } ] } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api delete-bucket \ --bucket=tempbucket $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-buckets { "Buckets": [ { "Name": "testbucket", "CreationDate": "2022-05-03T11:32:34.156000+00:00" } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET http://localhost:39999/api/v1/s3/ HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:43:20 GMT Content-Type: application/xml Content-Length: 254 Server: Jetty(9.4.43.v20210629) <ListAllMyBucketsResult> <Buckets> <Bucket> <Name>tempbucket</Name> <CreationDate>2022-05-03T14:43:03.651Z</CreationDate> </Bucket> <Bucket> <Name>testbucket</Name> <CreationDate>2022-05-03T14:34:56.420Z</CreationDate> </Bucket> </Buckets> </ListAllMyBucketsResult> $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X DELETE http://localhost:39999/api/v1/s3/tempbucket HTTP/1.1 204 No Content Date: Tue, 03 May 2022 21:43:25 GMT Server: Jetty(9.4.43.v20210629) $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET http://localhost:39999/api/v1/s3/ HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:43:28 GMT Content-Type: application/xml Content-Length: 161 Server: Jetty(9.4.43.v20210629) <ListAllMyBucketsResult> <Buckets> <Bucket> <Name>testbucket</Name> <CreationDate>2022-05-03T14:34:56.420Z</CreationDate> </Bucket> </Buckets> </ListAllMyBucketsResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs deletebuckettagging %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api get-bucket-tagging \ --bucket=testbucket { "TagSet": [ { "Key": "key1", "Value": "val1" }, { "Key": "key2", "Value": "val2" } ] } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api delete-bucket-tagging \ --bucket=testbucket $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api get-bucket-tagging \ --bucket=testbucket { "TagSet": [] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket?tagging" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:31:07 GMT Content-Type: application/xml Content-Length: 124 Server: Jetty(9.4.43.v20210629) <Tagging> <TagSet> <Tag> <Key>key1</Key> <Value>val1</Value> </Tag> <Tag> <Key>key2</Key> <Value>val2</Value> </Tag> </TagSet> </Tagging> $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X DELETE "http://localhost:39999/api/v1/s3/testbucket?tagging" HTTP/1.1 204 No Content Date: Tue, 03 May 2022 23:32:26 GMT Server: Jetty(9.4.43.v20210629) $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket?tagging" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:32:27 GMT Content-Type: application/xml Content-Length: 28 Server: Jetty(9.4.43.v20210629) <Tagging><TagSet/></Tagging> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs delete_object %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-objects-v2 \ --bucket=testbucket { "Contents": [ { "Key": "temp.txt", "LastModified": "2022-05-03T11:55:01.925000+00:00", "Size": 27040 }, { "Key": "test.txt", "LastModified":

"Size": 27040 } ] } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api delete-object \ --bucket=testbucket --key=temp.txt $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-objects-v2 \ --bucket=testbucket { "Contents": [ { "Key": "test.txt", "LastModified": "2022-05-03T11:55:01.925000+00:00", "Size": 27040 } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET http://localhost:39999/api/v1/s3/testbucket HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:59:27 GMT Content-Type: application/xml Content-Length: 540 Server: Jetty(9.4.43.v20210629) <ListBucketResult> <version2>false</version2> <Contents> <Key>temp.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:50:07.790Z</LastModified> </Contents> <Contents> <Key>test.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:47:36.600Z</LastModified> </Contents> <Marker/> <IsTruncated>false</IsTruncated> <Prefix/> <Name>testbucket</Name> <MaxKeys>1000</MaxKeys> <EncodingType>url</EncodingType> </ListBucketResult> $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X DELETE http://localhost:39999/api/v1/s3/testbucket/temp.txt HTTP/1.1 204 No Content Date: Tue, 03 May 2022 22:01:56 GMT Server: Jetty(9.4.43.v20210629) $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET http://localhost:39999/api/v1/s3/testbucket HTTP/1.1 200 OK Date: Tue, 03 May 2022 22:01:59 GMT Content-Type: application/xml Content-Length: 318 Server: Jetty(9.4.43.v20210629) <ListBucketResult> <version2>false</version2> <Contents> <Key>test.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:47:36.600Z</LastModified> </Contents> <Marker/> <IsTruncated>false</IsTruncated> <Prefix/> <Name>testbucket</Name> <MaxKeys>1000</MaxKeys> <EncodingType>url</EncodingType> </ListBucketResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs delete_objects %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-objects-v2 \ --bucket=tempbucket { "Contents": [ { "Key": "foo.txt", "LastModified": "2022-05-03T11:57:00.767000+00:00", "Size": 27040 }, { "Key": "temp.txt", "LastModified": "2022-05-03T11:56:11.245000+00:00", "Size": 27040 }, { "Key": "temp2.txt", "LastModified": "2022-05-03T11:56:31.414000+00:00", "Size": 27040 } ] } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api delete-objects \ --bucket=tempbucket --delete="Objects=[{Key=temp.txt},{Key=temp2.txt}]" $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-objects-v2 \ --bucket=tempbucket { "Contents": [ { "Key": "foo.txt", "LastModified": "2022-05-03T11:57:00.767000+00:00", "Size": 27040 } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET http://localhost:39999/api/v1/s3/tempbucket HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:59:27 GMT Content-Type: application/xml Content-Length: 540 Server: Jetty(9.4.43.v20210629) <ListBucketResult> <version2>false</version2> <Contents> <Key>foo.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:59:05.906Z</LastModified> </Contents> <Contents> <Key>temp.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:58:58.204Z</LastModified> </Contents> <Contents> <Key>temp2.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:59:01.987Z</LastModified> </Contents> <Marker/> <IsTruncated>false</IsTruncated> <Prefix/> <Name>tempbucket</Name> <MaxKeys>1000</MaxKeys> <EncodingType>url</EncodingType> </ListBucketResult> $ cat delete.xml <Delete xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Object> <Key>temp.txt</Key> </Object> <Object> <Key>temp2.txt</Key> </Object> <Quiet>false</Quiet> </Delete> $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -H "Content-Type: application/xml" \ -X POST --data "@delete.xml" "http://localhost:39999/api/v1/s3/testbucket?delete" HTTP/1.1 200 OK Date: Tue, 03 May 2022 22:56:17 GMT Content-Type: application/xml Content-Length: 208 Server: Jetty(9.4.43.v20210629) <DeleteResult> <Deleted> <Key>temp2.txt</Key> <DeleteMarker/> <DeleteMarkerVersionId/> <VersionId/> </Deleted> <Deleted> <Key>temp.txt</Key> <DeleteMarker/> <DeleteMarkerVersionId/> <VersionId/> </Deleted> </DeleteResult> $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET http://localhost:39999/api/v1/s3/tempbucket HTTP/1.1 200 OK Date: Tue, 03 May 2022 22:28:31 GMT Content-Type: application/xml Content-Length: 317 Server: Jetty(9.4.43.v20210629) <ListBucketResult> <version2>false</version2> <Contents> <Key>foo.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:59:05.906Z</LastModified> </Contents> <Marker/> <IsTruncated>false</IsTruncated> <Prefix/> <Name>tempbucket</Name> <MaxKeys>1000</MaxKeys> <EncodingType>url</EncodingType> </ListBucketResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs deleteobjecttagging %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api get-object-tagging \ --bucket=testbucket --key=test.txt { "TagSet": [ { "Key": "key1", "Value": "val1" }, { "Key": "key2", "Value": "val2" } ] } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api delete-object-tagging \ --bucket=testbucket --key=test.txt $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api get-object-tagging \ --bucket=testbucket --key=test.txt { "TagSet": [] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket?tagging" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:31:07 GMT Content-Type: application/xml Content-Length: 124 Server:

<Tagging> <TagSet> <Tag> <Key>key1</Key> <Value>val1</Value> </Tag> <Tag> <Key>key2</Key> <Value>val2</Value> </Tag> </TagSet> </Tagging> $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X DELETE "http://localhost:39999/api/v1/s3/testbucket/test.txt?tagging" HTTP/1.1 204 No Content Date: Tue, 03 May 2022 23:37:46 GMT Server: Jetty(9.4.43.v20210629) $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket/test.txt?tagging" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:37:47 GMT Content-Type: application/octet-stream Content-Length: 28 Server: Jetty(9.4.43.v20210629) <Tagging><TagSet/></Tagging> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs getbuckettagging %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api get-bucket-tagging \ --bucket=testbucket { "TagSet": [ { "Key": "key1", "Value": "val1" }, { "Key": "key2", "Value": "val2" } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket?tagging" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:31:07 GMT Content-Type: application/xml Content-Length: 124 Server: Jetty(9.4.43.v20210629) <Tagging> <TagSet> <Tag> <Key>key1</Key> <Value>val1</Value> </Tag> <Tag> <Key>key2</Key> <Value>val2</Value> </Tag> </TagSet> </Tagging> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs get_object %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api get-object \ --bucket=testbucket --key=test.txt /tmp/test.txt { "LastModified": "2022-05-03T18:55:01+00:00", "ContentLength": 27040, "ETag": "0cc175b9c0f1b6a831c399e269772661", "ContentType": "application/octet-stream", "Metadata": {} } $ stat /tmp/test.txt File: /tmp/test.txt Size: 27040 Blocks: 56 IO Block: 4096 regular file ... ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET http://localhost:39999/api/v1/s3/testbucket/test.txt HTTP/1.1 200 OK Date: Tue, 03 May 2022 22:59:43 GMT Last-Modified: Tue, 03 May 2022 21:47:36 GMT ETag: 0cc175b9c0f1b6a831c399e269772661 Content-Type: application/octet-stream Content-Length: 27040 Server: Jetty(9.4.43.v20210629) ................. file contents ................. ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs getobjecttagging %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api get-object-tagging \ --bucket=testbucket --key=test.txt { "TagSet": [ { "Key": "key1", "Value": "val1" }, { "Key": "key2", "Value": "val2" } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket?tagging" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:31:07 GMT Content-Type: application/xml Content-Length: 124 Server: Jetty(9.4.43.v20210629) <Tagging> <TagSet> <Tag> <Key>key1</Key> <Value>val1</Value> </Tag> <Tag> <Key>key2</Key> <Value>val2</Value> </Tag> </TagSet> </Tagging> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs head_bucket %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api head-bucket \ --bucket=testbucket ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ --head http://localhost:39999/api/v1/s3/testbucket HTTP/1.1 200 OK Date: Tue, 15 Nov 2022 04:49:12 GMT Content-Type: application/xml Content-Length: 0 Server: Jetty(9.4.43.v20210629) ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs head_object %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api head-object \ --bucket=testbucket --key=test.txt { "LastModified": "2022-05-03T18:55:01+00:00", "ContentLength": 27040, "ETag": "0cc175b9c0f1b6a831c399e269772661", "ContentType": "application/octet-stream", "Metadata": {} } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ --head http://localhost:39999/api/v1/s3/testbucket/test.txt HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:54:22 GMT Last-Modified: Tue, 03 May 2022 21:47:36 GMT ETag: 0cc175b9c0f1b6a831c399e269772661 Content-Type: application/octet-stream Content-Length: 27040 Server:

``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs list_buckets %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-buckets { "Buckets": [ { "Name": "testbucket", "CreationDate": "2022-05-03T11:32:34.156000+00:00" } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET http://localhost:39999/api/v1/s3/ HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:35:23 GMT Content-Type: application/xml Content-Length: 161 Server: Jetty(9.4.43.v20210629) <ListAllMyBucketsResult> <Buckets> <Bucket> <Name>testbucket</Name> <CreationDate>2022-05-03T14:34:56.420Z</CreationDate> </Bucket> </Buckets> </ListAllMyBucketsResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs list_objects %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-objects \ --bucket=testbucket { "Contents": [ { "Key": "test.txt", "LastModified": "2022-05-03T11:35:59.243000+00:00", "Size": 27040 }, { "Key": "test_copy.txt", "LastModified": "2022-05-03T11:37:16.185000+00:00", "Size": 27040 } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET http://localhost:39999/api/v1/s3/testbucket HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:50:26 GMT Content-Type: application/xml Content-Length: 434 Server: Jetty(9.4.43.v20210629) <ListBucketResult> <version2>false</version2> <Contents> <Key>test.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:47:36.600Z</LastModified> </Contents> <Contents> <Key>test_copy.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:50:07.790Z</LastModified> </Contents> <Marker/> <IsTruncated>false</IsTruncated> <Prefix/> <Name>testbucket</Name> <MaxKeys>1000</MaxKeys> <EncodingType>url</EncodingType> </ListBucketResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs list_uploads %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint="http://localhost:39999/api/v1/s3" s3api list-multipart-uploads --bucket "testbucket" { "Uploads": [ { "UploadId": "c4cddf71-914a-4cee-b2af-8cfb7def7d04", "Key": "object", "Initiated": "2022-07-01T11:21:14.738000+00:00" }, { "UploadId": "6367cf96-ea4e-4447-b931-c5bc91200375", "Key": "object", "Initiated": "2022-07-01T11:18:25.290000+00:00" }, { "UploadId": "e111c33b-5c18-4ecd-b543-2849cdbbf22b", "Key": "object2", "Initiated": "2022-07-01T11:21:25.182000+00:00" } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket?uploads" HTTP/1.1 200 OK Date: Fri, 01 Jul 2022 18:23:43 GMT Content-Type: application/xml Content-Length: 499 Server: Jetty(9.4.46.v20220331) <ListMultipartUploadsResult> <Bucket>testbucket</Bucket> <Upload> <Key>object</Key> <UploadId>c4cddf71-914a-4cee-b2af-8cfb7def7d04</UploadId> <Initiated>2022-07-01T11:21:14.738Z</Initiated> </Upload> <Upload> <Key>object</Key> <UploadId>6367cf96-ea4e-4447-b931-c5bc91200375</UploadId> <Initiated>2022-07-01T11:18:25.290Z</Initiated> </Upload> <Upload> <Key>object2</Key> <UploadId>e111c33b-5c18-4ecd-b543-2849cdbbf22b</UploadId> <Initiated>2022-07-01T11:21:25.182Z</Initiated> </Upload> </ListMultipartUploadsResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs listobjectsv2 %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-objects-v2 \ --bucket=testbucket { "Contents": [ { "Key": "test.txt", "LastModified": "2022-05-03T11:35:59.243000+00:00", "Size": 27040 }, { "Key": "test_copy.txt", "LastModified": "2022-05-03T11:37:16.185000+00:00", "Size": 27040 } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket?list-type=2" HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:56:20 GMT Content-Type: application/xml Content-Length: 438 Server: Jetty(9.4.43.v20210629) <ListBucketResult> <version2>true</version2> <Contents> <Key>test.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:47:36.600Z</LastModified> </Contents> <Contents> <Key>test_copy.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:50:07.790Z</LastModified> </Contents> <IsTruncated>false</IsTruncated> <Prefix/> <KeyCount>2</KeyCount> <Name>testbucket</Name> <MaxKeys>1000</MaxKeys> <EncodingType>url</EncodingType> </ListBucketResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs list_parts %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-parts \ --bucket=testbucket --key=multipart.txt --upload-id=6367cf96-ea4e-4447-b931-c5bc91200375 { "Parts": [ { "PartNumber": 1, "LastModified": "2022-05-03T12:56:27.775000+00:00", "ETag": "", "Size": 27040 } ], "ChecksumAlgorithm": null, "Initiator": null, "Owner": null, "StorageClass": "STANDARD" } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket/multipart.txt?uploadId=6367cf96-ea4e-4447-b931-c5bc91200375" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:49:05 GMT Content-Type: application/octet-stream Content-Length: 314 Server: Jetty(9.4.43.v20210629) <ListPartsResult> <Bucket>/testbucket</Bucket> <Key>multipart.txt</Key> <UploadId>6367cf96-ea4e-4447-b931-c5bc91200375</UploadId> <StorageClass>STANDARD</StorageClass> <IsTruncated>false</IsTruncated> <Part> <PartNumber>1</PartNumber>

<ETag></ETag> <Size>27040</Size> </Part> </ListPartsResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs putbuckettagging %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api get-bucket-tagging \ --bucket=testbucket { "TagSet": [] } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api put-bucket-tagging \ --bucket=testbucket --tagging='TagSet=[{Key=key1,Value=val1},{Key=key2,Value=val2}]' $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api get-bucket-tagging \ --bucket=testbucket { "TagSet": [ { "Key": "key1", "Value": "val1" }, { "Key": "key2", "Value": "val2" } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket?tagging" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:30:25 GMT Content-Type: application/xml Content-Length: 28 Server: Jetty(9.4.43.v20210629) <Tagging><TagSet/></Tagging> $ cat tags.xml <Tagging> <TagSet> <Tag> <Key>key1</Key> <Value>val1</Value> </Tag> <Tag> <Key>key2</Key> <Value>val2</Value> </Tag> </TagSet> </Tagging> $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -H "Content-Type: application/xml" \ -X PUT "http://localhost:39999/api/v1/s3/testbucket?tagging" --data-binary "@tags.xml" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:31:05 GMT Content-Length: 0 Server: Jetty(9.4.43.v20210629) $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket?tagging" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:31:07 GMT Content-Type: application/xml Content-Length: 124 Server: Jetty(9.4.43.v20210629) <Tagging> <TagSet> <Tag> <Key>key1</Key> <Value>val1</Value> </Tag> <Tag> <Key>key2</Key> <Value>val2</Value> </Tag> </TagSet> </Tagging> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs put_object %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api put-object \ --bucket=testbucket --key=test.txt --body="${ALLUXIO_HOME}/LICENSE" { "ETag": "911df44b7ff57801ca8d74568e4ebfbe" } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-objects \ --bucket=testbucket { "Contents": [ { "Key": "test.txt", "LastModified": "2022-05-03T11:35:59.243000+00:00", "Size": 27040 } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X PUT http://localhost:39999/api/v1/s3/testbucket/test.txt -T "${ALLUXIO_HOME}/LICENSE" HTTP/1.1 100 Continue HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:47:36 GMT ETag: 911df44b7ff57801ca8d74568e4ebfbe Content-Length: 0 Server: Jetty(9.4.43.v20210629) $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET http://localhost:39999/api/v1/s3/testbucket HTTP/1.1 200 OK Date: Tue, 03 May 2022 21:47:44 GMT Content-Type: application/xml Content-Length: 318 Server: Jetty(9.4.43.v20210629) <ListBucketResult> <version2>false</version2> <Contents> <Key>test.txt</Key> <Size>27040</Size> <LastModified>2022-05-03T14:47:36.600Z</LastModified> </Contents> <Marker/> <IsTruncated>false</IsTruncated> <Prefix/> <Name>testbucket</Name> <MaxKeys>1000</MaxKeys> <EncodingType>url</EncodingType> </ListBucketResult> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs putobjecttagging %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api get-object-tagging \ --bucket=testbucket --key=test.txt { "TagSet": [] } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api put-object-tagging \ --bucket=testbucket --key=test.txt --tagging='TagSet=[{Key=key1,Value=val1},{Key=key2,Value=val2}]' $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api get-object-tagging \ --bucket=testbucket --key=test.txt { "TagSet": [ { "Key": "key1", "Value": "val1" }, { "Key": "key2", "Value": "val2" } ] } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket/test.txt?tagging" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:30:25 GMT Content-Type: application/xml Content-Length: 28 Server: Jetty(9.4.43.v20210629) <Tagging><TagSet/></Tagging> $ cat tags.xml <Tagging> <TagSet> <Tag> <Key>key1</Key> <Value>val1</Value> </Tag> <Tag> <Key>key2</Key> <Value>val2</Value> </Tag> </TagSet> </Tagging> $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -H "Content-Type: application/xml" \ -X PUT "http://localhost:39999/api/v1/s3/testbucket/test.txt?tagging" --data-binary "@tags.xml" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:35:28 GMT Content-Length: 0 Server: Jetty(9.4.43.v20210629) $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X GET "http://localhost:39999/api/v1/s3/testbucket/test.txt?tagging" HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:35:58 GMT Content-Type: application/octet-stream Content-Length: 126 Server:

<Tagging> <TagSet> <Tag> <Key>key1</Key> <Value>val1</Value> </Tag> <Tag> <Key>key2</Key> <Value>val2</Value> </Tag> </TagSet> </Tagging> ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs upload_part %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api upload-part \ --bucket=testbucket --key=multipart.txt --upload-id=6367cf96-ea4e-4447-b931-c5bc91200375 --part-number=1 --body="${ALLUXIO_HOME}/LICENSE" { "ETag": "911df44b7ff57801ca8d74568e4ebfbe" } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-parts \ --bucket=testbucket --key=multipart.txt --upload-id=6367cf96-ea4e-4447-b931-c5bc91200375 { "Parts": [ { "PartNumber": 1, "LastModified": "2022-05-03T12:56:27.775000+00:00", "ETag": "", "Size": 27040 } ], "ChecksumAlgorithm": null, "Initiator": null, "Owner": null, "StorageClass": "STANDARD" } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -X PUT -T "${ALLUXIO_HOME}/LICENSE" "http://localhost:39999/api/v1/s3/testbucket/multipart.txt?uploadId=6367cf96-ea4e-4447-b931-c5bc91200375&partNumber=1" HTTP/1.1 100 Continue HTTP/1.1 200 OK Date: Tue, 03 May 2022 23:51:19 GMT ETag: "911df44b7ff57801ca8d74568e4ebfbe" Content-Length: 0 Server: Jetty(9.4.43.v20210629) $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-parts \ --bucket=testbucket --key=multipart_copy.txt --upload-id=6367cf96-ea4e-4447-b931-c5bc91200375 { "Parts": [ { "PartNumber": 1, "LastModified": "2022-05-03T13:00:13.584000+00:00", "ETag": "", "Size": 27040 } ], "ChecksumAlgorithm": null, "Initiator": null, "Owner": null, "StorageClass": "STANDARD" } ``` {% endnavtab %} {% endnavtabs %} See {:target="_blank"} on AWS {% navtabs uploadpartcopy %} {% navtab AWS CLI %} ```shell $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api upload-part-copy \ --bucket=testbucket --key=object --upload-id=6367cf96-ea4e-4447-b931-c5bc91200375 --part-number=1 --copy-source=testbucket/object { "CopyPartResult": { "ETag": "0cc175b9c0f1b6a831c399e269772661" } } $ aws --profile alluxio-s3 --endpoint "http://localhost:39999/api/v1/s3/" s3api list-parts \ --bucket=testbucket --key=object --upload-id=6367cf96-ea4e-4447-b931-c5bc91200375 { "Parts": [ { "PartNumber": 1, "LastModified": "2022-07-01T11:27:48.942000+00:00", "ETag": "", "Size": 1 } ], "ChecksumAlgorithm": null, "Initiator": null, "Owner": null, "StorageClass": "STANDARD" } ``` {% endnavtab %} {% navtab REST Clients %} ```shell $ curl -i -H "Authorization: AWS4-HMAC-SHA256 Credential=testuser/... SignedHeaders=... Signature=..." \ -H "x-amz-copy-source: testbucket/object" \ -X PUT 'http://localhost:39999/api/v1/s3/testbucket/object?uploadId=6367cf96-ea4e-4447-b931-c5bc91200375&partNumber=1' HTTP/1.1 200 OK Date: Fri, 01 Jul 2022 18:31:34 GMT Content-Type: application/xml Content-Length: 78 Server: Jetty(9.4.46.v20220331) <CopyPartResult><ETag>0cc175b9c0f1b6a831c399e269772661</ETag></CopyPartResult> ``` {% endnavtab %} {% endnavtabs %} Tested for Python 2.7. Please note you have to install boto package first. ```shell $ pip install boto ``` ```python import boto import boto.s3.connection conn = boto.connect_s3( awsaccesskey_id = '', awssecretaccess_key = '', host = 'localhost', port = 39999, path = '/api/v1/s3', is_secure=False, calling_format = boto.s3.connection.OrdinaryCallingFormat(), ) ``` By default, authenticating with no accesskeyid uses the user that was used to launch the proxy as the user performing the file system actions. Set the ```awsaccesskey_id``` to a different username to perform the actions under a different user. ```python bucketName = 'bucket-for-testing' bucket = conn.create_bucket(bucketName) ``` Authenticating as a user is necessary to have buckets returned by this operation. ```python conn = boto.connect_s3( awsaccesskey_id = 'testuser', awssecretaccess_key = '', host = 'localhost', port = 39999, path = '/api/v1/s3', is_secure=False, calling_format = boto.s3.connection.OrdinaryCallingFormat(), ) conn.getallbuckets() ``` ```python smallObjectKey = 'small.txt' smallObjectContent = 'Hello World!' key = bucket.new_key(smallObjectKey) key.setcontentsfrom_string(smallObjectContent) ``` ```python assert smallObjectContent == key.getcontentsas_string() ``` Create a 8MB file on local file system. ```shell $ dd if=/dev/zero of=8mb.data bs=1048576 count=8 ``` Then use python S3 client to upload this as an object ```python largeObjectKey = 'large.txt' largeObjectFile = '8mb.data' key = bucket.new_key(largeObjectKey) with open(largeObjectFile, 'rb') as f: key.setcontentsfrom_file(f) with open(largeObjectFile, 'rb') as f: largeObject = f.read() ``` ```python assert largeObject == key.getcontentsas_string() ``` ```python bucket.delete_key(smallObjectKey) bucket.delete_key(largeObjectKey) ``` ```python mp = bucket.initiatemultipartupload(largeObjectKey) ``` ```python import math, os from filechunkio import FileChunkIO sourceSize = os.stat(largeObjectFile).st_size chunkSize = 1048576 chunkCount = int(math.ceil(sourceSize / float(chunkSize))) for i in range(chunkCount): offset = chunkSize * i bytes = min(chunkSize, sourceSize - offset) with FileChunkIO(largeObjectFile, 'r', offset=offset, bytes=bytes) as fp: mp.uploadpartfromfile(fp, partnum=i + 1) ``` ```python mp.complete_upload() ``` Non-completed uploads can be aborted. ```python mp.cancel_upload() ``` ```python bucket.delete_key(largeObjectKey) conn.delete_bucket(bucketName) ```