Project: Alluxio (Runtime / Cloud Native Storage)
File: CephFS.md

layout: global title: CephFS This guide describes how to configure Alluxio with {:target="_blank"} as the under storage system. The Ceph File System (CephFS) is a POSIX-compliant file system built on top of Cephs distributed object store, RADOS. CephFS endeavors to provide a state-of-the-art, multi-use, highly available, and performant file store for a variety of applications, including traditional use-cases like shared home directories, HPC scratch space, and distributed workflow shared storage. Alluxio supports two different implementations of under storage system for CephFS. Fore more information, please read its documentation: {:target="_blank"} {:target="_blank"} If you haven't already, please see before you get started. In preparation for using CephFS with Alluxio: <table class="table table-striped"> <tr> <td markdown="span" style="width:30%">`<CEPHFSCONFFILE>`</td> <td markdown="span">Local path to Ceph configuration file ceph.conf</td> </tr> <tr> <td markdown="span" style="width:30%">`<CEPHFS_NAME>`</td> <td markdown="span">Ceph URI that is used to identify dameon instances in the ceph.conf</td> </tr> <tr> <td markdown="span" style="width:30%">`<CEPHFS_DIRECTORY>`</td> <td markdown="span">The directory you want to use, either by creating a new directory or using an existing one</td> </tr> <tr> <td markdown="span" style="width:30%">`<CEPHFSAUTHID>`</td> <td markdown="span">Ceph user id</td> </tr> <tr> <td markdown="span" style="width:30%">`<CEPHFSKEYRINGFILE>`</td> <td markdown="span">Ceph keyring file that stores one or more Ceph authentication keys</td> </tr> </table> Follow {:target="_blank"} to install below packages: ``` cephfs-java libcephfs_jni libcephfs2 ``` ```shell $ ln -s /usr/lib64/libcephfsjni.so.1.0.0 /usr/lib64/libcephfsjni.so $ ln -s /usr/lib64/libcephfs.so.2.0.0 /usr/lib64/libcephfs.so $ java_path=`which java | xargs readlink | sed 's#/bin/java##g'` $ ln -s /usr/share/java/libcephfs.jar $java_path/jre/lib/ext/libcephfs.jar ``` ```shell $ curl -o $java_path/jre/lib/ext/hadoop-cephfs.jar -s https://download.ceph.com/tarballs/hadoop-cephfs.jar ``` To use CephFS as the UFS of Alluxio root mount point, you need to configure Alluxio to use under storage systems by modifying `conf/alluxio-site.properties`. If it does not exist, create the configuration file from the template. ```shell $ cp conf/alluxio-site.properties.template conf/alluxio-site.properties $ cp conf/core-site.xml.template conf/core-site.xml ``` Set the following property to define CephFS as the root mount ```properties alluxio.dora.client.ufs.root=cephfs://mon1\;mon2\;mon3/ ``` {% navtabs Setup %} {% navtab cephfs %} Modify `conf/alluxio-site.properties` to include: ```properties alluxio.underfs.cephfs.conf.file=<CEPHFSCONFFILE> alluxio.underfs.cephfs.mds.namespace=<CEPHFS_NAME> alluxio.underfs.cephfs.mount.point=<CEPHFS_DIRECTORY> alluxio.underfs.cephfs.auth.id=<CEPHFSAUTHID> alluxio.underfs.cephfs.auth.keyring=<CEPHFSKEYRINGFILE> ``` {% endnavtab %} {% navtab cephfs-hadoop %} Modify `conf/alluxio-site.properties` to include: ```properties alluxio.underfs.hdfs.configuration=${ALLUXIO_HOME}/conf/core-site.xml ``` Modify `conf/core-site.xml` to include: ```xml <configuration> <property> <name>fs.default.name</name> <value>ceph://mon1,mon2,mon3/</value> </property> <property> <name>fs.defaultFS</name> <value>ceph://mon1,mon2,mon3/</value> </property> <property> <name>ceph.data.pools</name> <value>${data-pools}</value> </property> <property> <name>ceph.auth.id</name> <value>${client-id}</value> </property> <property> <name>ceph.conf.options</name> <value>clientmountgid=${gid},clientmountuid=${uid},clientmdsnamespace=${ceph-fs-name}</value> </property> <property> <name>ceph.root.dir</name> <value>${ceph-fs-dir}</value> </property> <property> <name>ceph.mon.address</name> <value>mon1,mon2,mon3</value> </property> <property> <name>fs.AbstractFileSystem.ceph.impl</name> <value>org.apache.hadoop.fs.ceph.CephFs</value> </property> <property> <name>fs.ceph.impl</name> <value>org.apache.hadoop.fs.ceph.CephFileSystem</value> </property> <property> <name>ceph.auth.keyring</name> <value>${client-keyring-file}</value> </property> </configuration> ``` {% endnavtab %} {% endnavtabs %} Once you have configured Alluxio to CephFS, try to see that everything works. ```shell $ ./bin/alluxio init format $ ./bin/alluxio process start local ``` Run a simple example program: ```shell $ ./bin/alluxio exec basicIOTest ``` Visit your cephfs to verify the files and directories created by Alluxio exist. You should see files named like: ``` ${cephfs-dir}/defaulttestsfiles/BasicCACHETHROUGH ``` In cephfs, you can visit cephfs with ceph-fuse or mount by POSIX APIs. {:target="_blank"} In Alluxio, you can visit the nested directory in the Alluxio. Alluxio's can be used for this purpose. ``` /mnt/cephfs/defaulttestsfiles/BasicCACHETHROUGH ``` CephFS and CephFS-Hadoop UFS integration is contributed and maintained by the Alluxio community. The source code for CephFS is located {:target="blank"} and for CephFS-Hadoop is located {:target="blank"}. Feel free submit pull requests to improve the integration and update the documentation {:target="_blank"} if any information is missing or out of date.