Project: Alluxio (Runtime / Cloud Native Storage)
File: Huawei-OBS.md

layout: global title: Huawei Object Storage Service This guide describes the instructions to configure {:target="_blank"} as Alluxio's under storage system. Huawei Object Storage Service (OBS) is a scalable service that provides secure, reliable, and cost-effective cloud storage for massive amounts of data. OBS provides unlimited storage capacity for objects of any format, catering to the needs of common users, websites, enterprises, and developers. For more information about Huawei OBS, please read its {:target="_blank"}. If you haven't already, please see before you get started. In preparation for using Huawei OBS with Alluxio: <table class="table table-striped"> <tr> <td markdown="span" style="width:30%">`<OBS_BUCKET>`</td> <td markdown="span">{:target="_blank"} or use an existing bucket</td> </tr> <tr> <td markdown="span" style="width:30%">`<OBS_DIRECTORY>`</td> <td markdown="span">The directory you want to use in the bucket, either by creating a new directory or using an existing one</td> </tr> <tr> <td markdown="span" style="width:30%">`<OBSACCESSKEY>`</td> <td markdown="span">Used to authenticate the identity of a requester. See {:target="_blank"}</td> </tr> <tr> <td markdown="span" style="width:30%">`<OBSSECRETKEY>`</td> <td markdown="span">Used to authenticate the identity of a requester. See {:target="_blank"}</td> </tr> <tr> <td markdown="span" style="width:30%">`<OBS_ENDPOINT>`</td> <td markdown="span">Domain name to access OBS in a region and is used to process requests of that region. See {:target="_blank"}</td> </tr> </table> To use Huawei OBS as the UFS of Alluxio root mount point, you need to configure Alluxio to use under storage systems by modifying `conf/alluxio-site.properties`. If it does not exist, create the configuration file from the template. ```shell $ cp conf/alluxio-site.properties.template conf/alluxio-site.properties ``` Specify an existing OBS bucket and directory as the underfs addresss system by modifying `conf/alluxio-site.properties` to include: ```properties alluxio.dora.client.ufs.root=obs://<OBSBUCKET>/<OBSDIRECTORY> ``` Note that if you want to mount the whole obs bucket, add a trailing slash after the bucket name (e.g. `obs://OBS_BUCKET/`). Specify credentials for OBS access by setting `fs.obs.accessKey` and `fs.obs.secretKey` in `alluxio-site.properties`. ```properties fs.obs.accessKey=<OBSACCESSKEY> fs.obs.secretKey=<OBSSECRETKEY> ``` Specify the OBS region by setting `fs.obs.endpoint` in `alluxio-site.properties` (e.g. obs.cn-north-4.myhuaweicloud.com). ```properties fs.obs.endpoint=<OBS_ENDPOINT> ``` Once you have configured Alluxio to Azure Blob Store, try to see that everything works. The default upload method uploads one file completely from start to end in one go. We use multipart-upload method to upload one file by multiple parts, every part will be uploaded in one thread. It won't generate any temporary files while uploading. To enable OBS multipart upload, you need to modify `conf/alluxio-site.properties` to include: ```properties alluxio.underfs.obs.multipart.upload.enabled=true ``` There are other parameters you can specify in `conf/alluxio-site.properties` to make the process faster and better. ```properties alluxio.underfs.object.store.multipart.upload.timeout ``` ```properties alluxio.underfs.obs.multipart.upload.threads ``` ```properties alluxio.underfs.obs.multipart.upload.partition.size ``` Huawei OBS UFS integration is contributed and maintained by the Alluxio community. The source code is located {:target="_blank"}. Feel free submit pull requests to improve the integration and update the documentation {:target="_blank"} if any information is missing or out of date.