Mount a Remote S3 Object Storage as Local Filesystem with S3FS-FUSE
By: Remko De Knikker
Tags: Glusterfs, Fuse, Storage, S3, NoSQL

In this tutorial, you will mount a remote Object Storage bucket using s3fs onto your local machine as a local directory. This practice is known as Filesystem in Userspace or FUSE . In this tutorial I used a MacOS (to be precise I used a MacOS Catalina version 10.15, MacBook Pro) and an Ubuntu environment at Labs.CognitiveClass.ai.

Filesystem in Userspace (FUSE) lets non-privileged users create a file system in their user space. The FUSE project consists of two components: a FUSE kernel module that is part of the Linux kernel since version 2.6.14, and the libfuse userspace library. The libfuse library provides a reference implementation for communication with the FUSE kernel module, providing client functions to mount the file system, unmount it, read requests from the kernel, and send responses back. FUSE is particularly useful for writing Virtual File Systems (VFS).

s3fs

s3fs or s3fs-fuse is an Amazon S3 (Simple Storage Service) and S3-based object stores compatible FUSE filesystem disk management utility that supports a subset of Single UNIX Specification (SUS) or POSIX including reading/writing files, directories, symlinks, mode, uid/gid, and extended attributes, while preserving the original file format, e.g. a plain text or MS Word document.

Applications that need to read and write to an NFS-style filesystem can use s3fs. It easily integrates applications with S3 compatible storage like IBM Cloud Object Storage. s3fs also allows you to interact with your cloud storage using familiar shell commands, like ls for listing or cp to copy files.

Performance is not equal to a true local filesystem, but you can use some advanced options to increase throughput. Object storage services have high-latency for time to first byte and lack `random write` access (requires rewriting the full object). Workloads that only read big files, like deep learning workloads, can achieve good throughput using s3fs.

Connect to IBM Cloud

For the tutorial you need to install the IBM Cloud CLI ,

You have to be logged in to your IBM Cloud account,

ibmcloud login -u <IBMId>

If you are using Single Sign-On (SSO) use the -sso flag to log in.

To create a free Object Storage instance in the UI,

Select the free Lite plan, and name your Object Storage instance, e.g. <username>-cos-1 ,

To create an Object Storage instance via the CLI,

COS_NAME=<username>-cos-1

RESOURCEGROUP=default ibmcloud resource groups

ibmcloud resource group-create $RESOURCEGROUP ibmcloud resource service-instance-create $COS_NAME cloud-object-storage Lite global -g $RESOURCEGROUP

In the UI, go to the Buckets menu item in the left menu.

Click Customize your bucket , and for regional select us-south , then click Create bucket .

Next, create credentials with HMAC keys for your Object Storage instance. Create an environment variable COS_CREDENTIALS with the name of the credentials,

COS_CREDENTIALS= cos-credentials-1 ibmcloud resource service-key-create $COS_CREDENTIALS Writer --instance-name $COS_NAME --parameters '{"HMAC":true}'

This will create credentials including among other HMAC keys, which we need later to authenticate the S3FS mount,

“cos_hmac_keys”: {

“access_key_id”: “c407e90c41c3463b8e7722048aa48edc”,

“secret_access_key”: “0f8c2cb6ef82c63d8d1f935a8ef6a87fe1bc16ed1ba8483c”

},

List the Object Storage instances,

ibmcloud cos config list

You will see that the CRN is not properly configured. Create a new environment variable CRN , and configure the Cloud Resource Name (CRN),

COS_CRN=$(ibmcloud resource service-key $COS_CREDENTIALS --output json | jq -r '.[0].credentials.resource_instance_id')

echo $COS_CRN ibmcloud cos config crn --crn $COS_CRN

To create a bucket via the CLI, set an environment variable with the bucket name, the bucket name must be globally unique,

COS_BUCKET_NAME=<cos-bucket-name>

Create a new bucket,

ibmcloud cos bucket-create --bucket $COS_BUCKET_NAME --class Standard --ibm-service-instance-id $COS_CRN

Then, in your client terminal Upload an Object to Object Storage

Next, upload a document to your bucket in the IBM Cloud Object Storage instance.

Create a new document,

COS_OBJECT_KEY=helloworld.txt

echo “Hello World! Today is $(date)” > $COS_OBJECT_KEY

Upload an object using S3Manager,

ibmcloud cos upload --bucket $COS_BUCKET_NAME --key $COS_OBJECT_KEY --file ./helloworld.txt --content-language en-US --content-type "text/plain" OK

Successfully uploaded object ‘helloworld.txt’ to bucket ‘e59a327194-cos-1-bucket1’.

Install s3fs

Install s3fs , on MacOS,

brew install s3fs

On Ubuntu,

sudo apt update

sudo apt install s3fs

Next, using the credentials we created earlier, create environment variables using the HMAC keys, and create an S3FS password file,

COS_ACCESS_KEY=$(ibmcloud resource service-key $COS_CREDENTIALS — output json | jq -r '.[0].credentials.cos_hmac_keys.access_key_id') COS_SECRET_KEY=$(ibmcloud resource service-key $COS_CREDENTIALS — output json | jq -r '.[0].credentials.cos_hmac_keys.secret_access_key') COS_S3FS_PASSWORD_FILE=s3fs-passwd echo $COS_ACCESS_KEY:$COS_SECRET_KEY > $COS_S3FS_PASSWORD_FILE chmod 0600 $COS_S3FS_PASSWORD_FILE

Mount Local File System

Mount the local directory using S3FS, in my case I create an Object Storage in the us-south region, which has a corresponding public endpoint including a us-south subdomain s3.us-south.cloud-object-storage.appdomain.cloud .

You can also find the public endpoint by browsing to your Object Storage instance, select your bucket, and go to configuration ,

Create a local directory and mount the directory to your remote object storage bucket.

mkdir cos_data LOCAL_MOUNTPOINT=$(pwd)/cos_data COS_PUBLIC_ENDPOINT=s3.us-south.cloud-object-storage.appdomain.cloud s3fs $COS_BUCKET_NAME -o passwd_file=$(pwd)/$COS_S3FS_PASSWORD_FILE -o url=https://$COS_PUBLIC_ENDPOINT $LOCAL_MOUNTPOINT

You should see the content of your IBM Cloud Object Storage, e.g using the Finder on macos.

Or using the cli on macos.

References

https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-s3fs