Project: Alluxio (Runtime / Cloud Native Storage)
File: Aliyun-OSS.md

layout: global title: Aliyun Object Storage Service This guide describes how to configure {:target="_blank"} as Alluxio's under storage system. Aliyun Object Storage Service (OSS) is a massive, secure and highly reliable cloud storage service provided by Alibaba Cloud. OSS provides multiple storage classes to help you manage and reduce storage costs. For more information about Aliyun OSS, please read its {:target="_blank"} If you haven't already, please see before you get started. In preparation for using OSS with Alluxio: <table class="table table-striped"> <tr> <td markdown="span" style="width:30%">`<OSS_BUCKET>`</td> <td markdown="span">{:target="_blank"} or use an existing bucket</td> </tr> <tr> <td markdown="span" style="width:30%">`<OSS_DIRECTORY>`</td> <td markdown="span">The directory you want to use in the bucket, either by creating a new directory or using an existing one</td> </tr> <tr> <td markdown="span" style="width:30%">`<OSSACCESSKEY_ID>`</td> <td markdown="span">ID used to identify a user. See {:target="_blank"}</td> </tr> <tr> <td markdown="span" style="width:30%">`<OSSACCESSKEY_SECRET>`</td> <td markdown="span">Secret is used to verify the identify of the user. See {:target="_blank"}</td> </tr> <tr> <td markdown="span" style="width:30%">`<OSS_ENDPOINT>`</td> <td markdown="span">Endpoints are the domain names that other services can use to access OSS. See {:target="_blank"}</td> </tr> </table> To use Aliyun OSS as the UFS of Alluxio root mount point, you need to configure Alluxio to use under storage systems by modifying `conf/alluxio-site.properties`. If it does not exist, create the configuration file from the template. ```shell $ cp conf/alluxio-site.properties.template conf/alluxio-site.properties ``` Specify an OSS bucket and directory as the underfs address by modifying `conf/alluxio-site.properties`. For example, the under storage address can be `oss://alluxio-bucket/` if you want to mount the whole bucket to Alluxio, or `oss://alluxio-bucket/alluxio/data` if only the directory `/alluxio/data` inside the oss bucket `alluxio-bucket` is mapped to Alluxio. ```properties alluxio.dora.client.ufs.root=oss://<OSSBUCKET>/<OSSDIRECTORY> ``` Specify credentials for Aliyun OSS access by adding the following properties in `conf/alluxio-site.properties`: ```properties fs.oss.accessKeyId=<OSSACCESSKEY_ID> fs.oss.accessKeySecret=<OSSACCESSKEY_SECRET> fs.oss.endpoint=<OSS_ENDPOINT> ``` Once you have configured Alluxio to Aliyun OSS, try to see that everything works. The default upload method uploads one file completely from start to end in one go. We use multipart-upload method to upload one file by multiple parts, every part will be uploaded in one thread. It won't generate any temporary files while uploading. To enable OSS multipart upload, you need to modify `conf/alluxio-site.properties` to include: ```properties alluxio.underfs.oss.multipart.upload.enabled=true ``` There are other parameters you can specify in `conf/alluxio-site.properties` to make the process faster and better. ```properties alluxio.underfs.object.store.multipart.upload.timeout ``` ```properties alluxio.underfs.oss.multipart.upload.threads ``` ```properties alluxio.underfs.oss.multipart.upload.partition.size ``` Aliyun OSS UFS integration is contributed and maintained by the Alluxio community. The source code is located {:target="_blank"}. Feel free submit pull requests to improve the integration and update the documentation {:target="_blank"} if any information is missing or out of date.