Package: ember-csi
Ember CSIMulti-vendor CSI plugin driver supporting over 80 storage drivers in a single plugin to provideblockandmountstorage to Container Orchestration systems.Free software: Apache Software License 2.0Documentation: PendingFeaturesThis CSI driver is up to date with latest CSI specs including thenew snapshots featurerecently introduced.Currently supported features are:Create block volumeCreating snapshotsCreating a block volume from a snapshotDelete block volumeDeleting snapshotsListing volumes with paginationListing snapshots with paginationAttaching volumesDetaching volumesReporting storage capacityProbing the nodeRetrieving the plugin infoRuntime DependenciesThis driver requires that Cinder v11.0 (OSP-12/Pike) is already installed in the system, how this is accomplished is left to the installer, as there are multiple ways this can be accomplished:From OSP repositoriesFrom RDO repositoriesFrom githubFrom other repositoriesAny other basic requirement is already handled byember-csiwhen installing from PyPi.Besides the basic dependencies there are also some drivers that have additional requirements that must be met for proper operation of the driver and/or attachment/detachment operations, just like in Cinder.Some of these Python dependencies for the Controller servicer are:DRBD: dbus and drbdmanageHPE 3PAR: python-3parclientKaminario: krestPure: purestorageDell EMC VMAX, IBM DS8K: pyOpenSSLHPE Lefthad: python-lefthandclientFujitsu Eternus DX: pywbemIBM XIV: pyxcliRBD: rados and rbdDell EMC VNX: storopsViolin: vmemclientINFINIDAT: infinisdk, capacity, infy.dtypes.wwn, infi.dtypes.iqnOther backends may also require additional packages, for example LVM on CentOS/RHEL requires thetargetclipackage, so please check with your hardware vendor.Besides the Controller requirements there are usually requirements for the Node servicer needed to handle the attaching and detaching of volumes to the node based on the connection used to access the storage.  For example:iSCSI: iscsi-initiator-tools and device-mapper-multipathRBD/Ceph: ceph-common packageInstallationFirst we need to install the Cinder Python package, for example to install from RDO on CentOS:$ sudo yum install -y centos-release-openstack-pike
    $ sudo yum install -y openstack-cinder python-pipThen we just need to install theember-csipackage:$ sudo pip install ember-csiNow we should install any additional package required by our backend.For iSCSI backends we'll want to install:$ sudo yum install iscsi-initiator-utils
    $ sudo yum install device-mapper-multipath
    $ sudo mpathconf --enable --with_multipathd y --user_friendly_names n --find_multipaths yFor RBD we'll also need a specific package:$ sudo yum install ceph-commonConfigurationThe CSI driver is configured via environmental variables, any value that doesn't have a default is a required value.NameRoleDescriptionDefaultExampleCSI_ENDPOINTallIP and port to bind the service[::]:50051192.168.1.22:50050CSI_MODEallRole the service should perform: controller, node, allallcontrollerX_CSI_SPEC_VERSIONallCSI Spec version to run. Supported v0.2 and v1.0v0.2.00.2.0X_CSI_STORAGE_NW_IPnodeIP address in the Node used to connect to the storageIP resolved from Node's fqdn192.168.1.22X_CSI_NODE_IDnodeID used by this node to identify itself to the controllerNode's fqdncsi_test_nodeX_CSI_PERSISTENCE_CONFIGallConfiguration of thecinderlibmetadata persistence plugin.{"storage": "crd", "namespace": "default"}{"storage": "db", "connection": "mysql+pymysql://root:stackdb@192.168.1.1/cinder?charset=utf8"}X_CSI_EMBER_CONFIGallGlobalEmberandcinderlibconfiguration{"project_id": "ember-csi.io", "user_id": "ember-csi.io", "root_helper": "sudo", "request_multipath": false, "plugin_name": "", "file_locks_path": "/var/lib/ember-csi/locks", "name": "io.ember-csi", "grpc_workers": 30, "enable_probe": false, "slow_operations: true, "disabled": []}{"project_id":"k8s project","user_id":"csi driver","root_helper":"sudo","plugin_name":"external-ceph","disabled":["snapshot","clone"]}X_CSI_BACKEND_CONFIGcontrollerDriver configuration{"name": "rbd", "driver": "RBD", "rbd_user": "cinder", "rbd_pool": "volumes", "rbd_ceph_conf": "/etc/ceph/ceph.conf", "rbd_keyring_conf": "/etc/ceph/ceph.client.cinder.keyring"}X_CSI_DEFAULT_MOUNT_FSnodeDefault mount filesystem when missing in publish callsext4btrfsX_CSI_SYSTEM_FILESallAll required storage driver-specific files archived in tar, tar.gz or tar.bz2 format/path/to/etc-ceph.tar.gzX_CSI_DEBUG_MODEallDebug mode (rpdb, pdb) to use. Disabled by default.rpdbX_CSI_ABORT_DUPLICATESallIf we want to abort or queue (default) duplicated requests.falsetrueThe only role that has been tested at the moment is the default one, where Controller and Node servicer are executed in the same service (CSI_MODE=all), and other modes are expected to have issues at the moment.The X_CSI_SYSTEM_FILES variable should point to a tar/tar.gz/tar.bz2 file accessible in the Ember CSI driver's filesystem. The contents of the archive will be extracted into '/'. A trusted user such as an operator/administrator with privileged access must create the archive before starting the driver.e.g.$ tar cvf ceph-files.tar /etc/ceph/ceph.conf /etc/ceph/ceph.client.cinder.keyring
tar: Removing leading `/' from member names
/etc/ceph/ceph.conf
/etc/ceph/ceph.client.cinder.keyring
$ export X_CSI_SYSTEM_FILES=`pwd`/ceph-files.tarStarting the pluginOnce we have installedember-csiand required dependencies (for the backend and for the connection type) we just have to run theember-csiservice with a user that can do passwordless sudo:$ ember-csiTesting the pluginThere are several examples of running the Ember CSI plugin in theexamplesdirectory both for a baremetal deployment and a containerized version of the driver.In all cases we have to run the plugin first before we can test it, and for that we have to check the configuration provided as a test before starting the plugin.  By default all examples run the service on port 50051.BaremetalFor example to test with the LVM driver on our development environment we can just run the following commands from the root of theember-csiproject:Note: The iscsi IP addresses are auto-assigned in thelvmenv file. You may change these IP addresses if desired:$ cd tmp
    $ sudo dd if=/dev/zero of=ember-volumes bs=1048576 seek=22527 count=1
    $ lodevice=`sudo losetup --show -f ./ember-volumes`
    $ sudo pvcreate $lodevice
    $ sudo vgcreate ember-volumes $lodevice
    $ sudo vgscan --cache
    $ cd ../examples/baremetal
    $ ./run.sh lvm
    py27 develop-inst-nodeps: /home/geguileo/code/ember-csi
    py27 installed: ...
    ___ summary ___
      py27: skipped tests
      congratulations :)
    Starting Ember CSI v0.0.2 (cinderlib: v0.2.1, cinder: v11.1.2.dev5, CSI spec: v0.2.0)
    Supported filesystems are: fat, ext4dev, vfat, ext3, ext2, msdos, ext4, hfsplus, cramfs, xfs, ntfs, minix, btrfs
    Running backend LVMVolumeDriver v3.0.0
    Debugging is OFF
    Now serving on [::]:50051...There is also an example of testing a Ceph cluster using a user called "cinder" and the "volumes" pool.  For the Ceph/RBD backend, due to a limitation in Cinder, we need to have both the credentials and the configuration in/etc/cephfor it to work:$ cd examples/baremetal
    $ ./run.sh rbd
    Starting Ember CSI v0.0.2 (cinderlib: v0.2.1, cinder: v11.1.2.dev5, CSI spec: v0.2.0)
    Supported filesystems are: fat, ext4dev, vfat, ext3, ext2, msdos, ext4, hfsplus, cramfs, xfs, ntfs, minix, btrfs
    Running backend LVMVolumeDriver v3.0.0
    Debugging is OFF
    Now serving on [::]:50051...There is also an XtremIO example that only requires the iSCSI connection packages.ContainerizedThere is a sampleDockerfileincluded in the project that has been used to create theakrog/ember-csicontainer available in the docker hub.There are two bash scripts, one for each example, that will run the CSI driver on a container, be aware that the container needs to run as privileged to mount the volumes.For the RBD example we need to copy our "ceph.conf" and "ceph.client.cinder.keyring" files, assuming we are using the "cinder" user into the example/docker directory replacing the existing ones:$ cd examples/docker
    $ ./rbd.sh
    Starting Ember CSI v0.0.2 (cinderlib: v0.2.1, cinder: v11.1.0, CSI spec: v0.2.0)
    Supported filesystems are: cramfs, minix, ext3, ext2, ext4, xfs, btrfs
    Running backend LVMVolumeDriver v3.0.0
    Debugging is ON with rpdb
    Now serving on [::]:50051...CSCNow that we have the service running we can use theCSC toolto run commands simulating the Container Orchestration system.Due to the recent changes in the CSI spec not all commands are available yet, so you won't be able to test the snapshot commands.Checking the plugin info:$ csc identity plugin-info -e tcp://127.0.0.1:50051
    "io.ember-csi"      "0.0.2" "cinder-driver"="RBDDriver"     "cinder-driver-supported"="True"        "cinder-driver-version"="1.2.0" "cinder-version"="11.1.0"       "cinderlib-version"="0.2.1"     "persistence"="DBPersistence"Checking the node id:$ csc node get-id -e tcp://127.0.0.1:50051
    localhost.localdomain

    $ hostname -f
    localhost.localdomainChecking the current backend capacity:$ csc controller get-capacity -e tcp://127.0.0.1:50051
    24202140712Creating a volume:$ csc controller create-volume --cap SINGLE_NODE_WRITER,block --req-bytes 2147483648 disk -e tcp://127.0.0.1:50051
    "5ee5fd7c-45cd-44cf-af7b-06081f680f2c"  2147483648Listing volumes:$ csc controller list-volumes -e tcp://127.0.0.1:50051
    "5ee5fd7c-45cd-44cf-af7b-06081f680f2c"  2147483648Store the volume id for all the following calls:$ vol_id=`csc controller list-volumes -e tcp://127.0.0.1:50051|awk '{ print gensub("\"","","g",$1)}'`Attaching the volume totmp/mnt/publishon baremetal as a block device:$ touch tmp/mnt/{staging,publish}

    $ csc controller publish --cap SINGLE_NODE_WRITER,block --node-id `hostname -f` $vol_id -e tcp://127.0.0.1:50051
    "5ee5fd7c-45cd-44cf-af7b-06081f680f2c"  "connection_info"="{\"connector\": {\"initiator\": \"iqn.1994-05.com.redhat:aa532823bac9\", \"ip\": \"127.0.0.1\", \"platform\": \"x86_64\", \"host\": \"localhost.localdomain\", \"do_local_attach\": false, \"os_type\": \"linux2\", \"multipath\": false}, \"conn\": {\"driver_volume_type\": \"rbd\", \"data\": {\"secret_uuid\": null, \"volume_id\": \"5ee5fd7c-45cd-44cf-af7b-06081f680f2c\", \"auth_username\": \"cinder\", \"secret_type\": \"ceph\", \"name\": \"volumes/volume-5ee5fd7c-45cd-44cf-af7b-06081f680f2c\", \"discard\": true, \"keyring\": \"[client.cinder]\\n\\tkey = AQCQPetaof03IxAAoHZJD6kGxiMQfLdn3QzdlQ==\\n\", \"cluster_name\": \"ceph\", \"hosts\": [\"192.168.1.22\"], \"auth_enabled\": true, \"ports\": [\"6789\"]}}}"

    $ csc node stage --pub-info connection_info="irrelevant" --cap SINGLE_NODE_WRITER,block --staging-target-path `realpath tmp/mnt/staging` $vol_id -e tcp://127.0.0.1:50051
    5ee5fd7c-45cd-44cf-af7b-06081f680f2c

    $ csc node publish --cap SINGLE_NODE_WRITER,block --pub-info connection_info="irrelevant" --staging-target-path `realpath tmp/mnt/staging` --target-path `realpath tmp/mnt/publish` $vol_id -e tcp://127.0.0.1:50051
    5ee5fd7c-45cd-44cf-af7b-06081f680f2cAttaching the volume totmp/mnt/publishon container as a block device:$ touch tmp/mnt/{staging,publish}

    $ csc controller publish --cap SINGLE_NODE_WRITER,block --node-id `hostname -f` $vol_id -e tcp://127.0.0.1:50051
    "5ee5fd7c-45cd-44cf-af7b-06081f680f2c"  "connection_info"="{\"connector\": {\"initiator\": \"iqn.1994-05.com.redhat:aa532823bac9\", \"ip\": \"127.0.0.1\", \"platform\": \"x86_64\", \"host\": \"localhost.localdomain\", \"do_local_attach\": false, \"os_type\": \"linux2\", \"multipath\": false}, \"conn\": {\"driver_volume_type\": \"rbd\", \"data\": {\"secret_uuid\": null, \"volume_id\": \"5ee5fd7c-45cd-44cf-af7b-06081f680f2c\", \"auth_username\": \"cinder\", \"secret_type\": \"ceph\", \"name\": \"volumes/volume-5ee5fd7c-45cd-44cf-af7b-06081f680f2c\", \"discard\": true, \"keyring\": \"[client.cinder]\\n\\tkey = AQCQPetaof03IxAAoHZJD6kGxiMQfLdn3QzdlQ==\\n\", \"cluster_name\": \"ceph\", \"hosts\": [\"192.168.1.22\"], \"auth_enabled\": true, \"ports\": [\"6789\"]}}}"

    $ csc node stage --pub-info connection_info="irrelevant" --cap SINGLE_NODE_WRITER,block --staging-target-path /mnt/staging $vol_id -e tcp://127.0.0.1:50051
    5ee5fd7c-45cd-44cf-af7b-06081f680f2c

    $ csc node publish --cap SINGLE_NODE_WRITER,block --pub-info connection_info="irrelevant" --staging-target-path /mnt/staging --target-path /mnt/publish $vol_id -e tcp://127.0.0.1:50051
    5ee5fd7c-45cd-44cf-af7b-06081f680f2cDetaching the volume on baremetal:$ csc node unpublish --target-path `realpath tmp/mnt/publish` $vol_id -e tcp://127.0.0.1:50051
    5ee5fd7c-45cd-44cf-af7b-06081f680f2c

    $ csc node unstage --staging-target-path `realpath tmp/mnt/staging` $vol_id -e tcp://127.0.0.1:50051
    5ee5fd7c-45cd-44cf-af7b-06081f680f2c

    $ csc controller unpublish --node-id `hostname -f` $vol_id -e tcp://127.0.0.1:50051
    5ee5fd7c-45cd-44cf-af7b-06081f680f2cDetaching the volume on container:$ csc node unpublish --target-path /mnt/publish $vol_id -e tcp://127.0.0.1:50051
    5ee5fd7c-45cd-44cf-af7b-06081f680f2c

    $ csc node unstage --staging-target-path /tmp/mnt/staging $vol_id -e tcp://127.0.0.1:50051
    5ee5fd7c-45cd-44cf-af7b-06081f680f2c

    $ csc controller unpublish --node-id `hostname -f` $vol_id -e tcp://127.0.0.1:50051
    5ee5fd7c-45cd-44cf-af7b-06081f680f2cDeleting the volume:$ csc controller delete-volume $vol_id -e tcp://127.0.0.1:50051If we want to use the mount interface instead of the block one, we can also do it making sure we create directories instead of files and replacing theblockword withmount,ext4if we want anext4filesystem.For example these would be the commands for the baremetal attach:$ mkdir tmp/mnt/{staging_dir,publish_dir}

    $ csc controller publish --cap SINGLE_NODE_WRITER,mount,ext4 --node-id `hostname -f` $vol_id -e tcp://127.0.0.1:50051

    $ csc node stage --pub-info connection_info="irrelevant" --cap SINGLE_NODE_WRITER,mount,ext4 --staging-target-path `realpath tmp/mnt/staging_dir` $vol_id -e tcp://127.0.0.1:50051
    5ee5fd7c-45cd-44cf-af7b-06081f680f2c

    $ csc node publish --pub-info connection_info="irrelevant" --cap SINGLE_NODE_WRITER,mount,ext4 -staging-target-path `realpath tmp/mnt/staging_dir` --target-path `realpath tmp/mnt/publish_dir` $vol_id -e tcp://127.0.0.1:50051
    5ee5fd7c-45cd-44cf-af7b-06081f680f2cCapable operational modesThe CSI spec defines a set ofAccessModesthat CSI drivers can support, such as single writer, single reader, multiple writers, single writer and multiple readers.This CSI driver currently only supportsSINGLE_MODE_WRITER, although it will also succeed with theSINGLE_MODE_READER_ONLYmode and mount it as read/write.DebuggingThe first tool for debugging is the log that displays detailed information on the driver code used byember-CSI.  We can enable INFO or DEBUG logs using theX_CSI_EMBER_CONFIGenvironmental variable.To enable logs, defaulting to INFO level, we must set thedisable_logskey tofalse.  If we want them at DEBUG levels, we also need to setdebugtotrue.For baremetal, enablig DEBUG log levels can be done like this:export X_CSI_EMBER_CONFIG={"project_id":"io.ember-csi","user_id":"io.ember-csi","root_helper":"sudo","plugin_name": "io.ember-csi","disable_logs":false,"debug":true}For containers we can just add the environmental variable to a file and import into our run using--env-fileor adding it to our command line with-e.In both cases it should not have theexportcommand:X_CSI_EMBER_CONFIG={"project_id":"io.ember-csi","user_id":"io.ember-csi","root_helper":"sudo","plugin_name": "io.ember-csi","disable_logs":false,"debug":true}Besides this basic debugging level, the Ember CSI plugin also supports live debugging when run in the baremetal and when running as a container.There are two mechanisms that can be used to debug the driver: withpdb, and withrpdb.The difference between them is thatpdbworks with stdin and stdout, whereasrpdbopens port 4444 to accept remote connections for debugging.Debugging the Ember CSI plugin requires enabling debugging on the plugin before starting it, and then one it is running we have to turn it on.Enabling debugging is done using theX_CSI_DEBUG_MODEenvironmental variable.  Setting it topdborrpdbwill enable debugging.  The plugin has this feature disabled by default, but ourlatestandmastercontainers have it enabled by default withrpdb.Once we have the plugin running with the debugging enable (we can see it in the start message) we can turn it on and off using theSIGUSR1signal, and the service will output the change with aDebugging is ONorDebugging is OFFmessage.After turning itONthe plugin will stop for debugging on the next GRPC request.  Going into interactive mode if usingpdbor opening port 4444 if usingrpdb.  When usingrpdbwe'll see the following message on the plugin:pdb is running on 127.0.0.1:4444Sending the signal to toggle ON/OFF the debugging is quite easy.  For baremetal we can do:$ pkill -USR1 ember-csiAnd for the container (assuming its namedember-csilike in the examples) we can do:$ docker kill -sUSR1 ember-csiIf we are usingrpdbthen we'll have to connect to the port:$ nc 127.0.0.1 4444TroubleshootingCSC commands timeoutIf you have a slow backend or a slow data network connection, and you are creating mount volumes, then you may run into "context deadline exceeded" errors when running the node staging command on the volume.This is just a 60 seconds timeout, and we can easily fix this by increasing allowed timeout for the command to complete.  For example to 5 minutes with-t5mor to 1 hour if we are manually debugging things on the server side with-t1h.Staging fails in container using iSCSIWhen I try to stage a volume using a containerizedNodeI see the error "ERROR root VolumeDeviceNotFound: Volume device not found at .".Turning the DEBUG log levels shows me login errors:2018-07-03 11:14:57.258 1 WARNING os_brick.initiator.connectors.iscsi [req-0e77bf32-a29b-40d1-b359-9e115435a94a io.ember-csi io.ember-csi - - -] Failed to connect to iSCSI portal 192.168.1.1:3260.
    2018-07-03 11:14:57.259 1 WARNING os_brick.initiator.connectors.iscsi [req-0e77bf32-a29b-40d1-b359-9e115435a94a io.ember-csi io.ember-csi - - -] Failed to login iSCSI target iqn.2008-05.com.something:smt00153500071-514f0c50023f6c01 on portal 192.168.1.1:3260 (exit code 12).: ProcessExecutionError: Unexpected error while running command.And looking into the host's journal (where theiscsiddaemon is running) I can seeKmoderrors:Jul 03 13:15:02 think iscsid[9509]: Could not insert module . Kmod error -2This seems to be cause by some kind of incompatibility between the host and the container's iSCSI modules.  We currently don't have a solution other than using a CentOS 7 host system.SupportFor any questions or concerns please file an issue with theember-csiproject or ping me on IRC (my handle is geguileo and I hang on the #openstack-cinder channel in Freenode).TODOThere are many things that need to be done in this POC driver, and here's a non exhaustive list:Support for NFS volumesSupport for Kubernetes CRDs as the persistence storageUnit testsFunctional testsImprove received parameters checkingMake driver more resilientTest driver in KubernetesReview some of the returned error codesSupport volume attributes via volume typesLook into multi-attachingSupport read-only modeReport capacity based on over provisioning valuesConfigure the private data locationHistory1.0.0 (2019-xx-yy)Bugs:Fix listings with invalid tokenFix list pagination with future tokensFix volume cloningFix getting volume stats0.9.0 (2019-06-04)Beta release with full CSI v0.2, v0.3, and v1.0 spec support.FeaturesMulti-driver support on single containerSupport for mount filesystemsSupport for blockTopology supportSnapshot supportLiveness probeCRD metadata persistence pluginMulti-version support on single containerAliases for configurationStorage driver list toolSupport live debugging of running driverDuplicated requests queuing support (for k8s)Support of mocked probeConfigurable default mount filesystemBugsFix issues receiving duplicated RPC callsFix UUID warningCheck staging and publishing targetsExit on binding error0.0.2 (2018-06-19)Use cinderlib v0.2.1 instead of github branch0.0.1 (2018-05-18)First release on PyPI.