Package: isilon-hadoop-tools
Isilon Hadoop ToolsTools for Using Hadoop with OneFSisilon_create_userscreates identities needed by Hadoop distributions compatible with OneFS.isilon_create_directoriescreates a directory structure with appropriate ownership and permissions in HDFS on OneFS.InstallationIsilon Hadoop Tools (IHT) currently requires Python 3.7+ and supports OneFS 8+.Python support schedules can be foundin the Python Developer's Guide.OneFS support schedules can be found in thePowerScale OneFS Product Availability Guide.Option 1: Install as a stand-alone command line tool.Usepipxto install IHT.pipxrequires Python 3.7 or later.Installpipx:python3-mpipinstall--userpipxTip: Newer versions of some Linux distributions (e.g.Debian 10,Ubuntu 19.04, etc.) offer native packages forpipx.python3-mpipxensurepathNote: You may need to restart your terminal for the$PATHupdates to take effect.Usepipxto installisilon_hadoop_tools:pipxinstallisilon_hadoop_toolsTest the installation:isilon_create_users--help
isilon_create_directories--helpUsepipxto uninstall at any time:pipxuninstallisilon_hadoop_toolsSee Python'sInstalling stand alone command line toolsguide for more information.Option 2: Create an ephemeral installation.Usepipto install IHT in a virtual environment.Python "Virtual Environments" allow Python packages to be installed in an isolated location for a particular application, rather than being installed globally.Use the built-invenvmodule to create a virtual environment:python3-mvenv./ihtInstallisilon_hadoop_toolsinto the virtual environment:iht/bin/pipinstallisilon_hadoop_toolsNote: This requires access to an up-to-date Python Package Index (PyPI, usuallyhttps://pypi.org/).
For offline installations, necessary resources can be downloaded to a USB flash drive which can be used instead:pip3download--dest/media/usb/iht-distsisilon_hadoop_toolsiht/bin/pipinstall--no-index--find-links/media/usb/iht-distsisilon_hadoop_toolsTest the installation:iht/bin/isilon_create_users--helpTip: Some users find it more convenient to "activate" the virtual environment (which prepends the virtual environment'sbin/to$PATH):sourceiht/bin/activate
isilon_create_users--help
isilon_create_directories--help
deactivateRemove the virtual environment to uninstall at any time:rm--recursiveiht/See Python'sInstalling Packagestutorial for more information.UsageTip:--helpcan be used with any IHT script to see extended usage information.To use IHT, you will need the following:$onefs, an IP address, hostname, or SmartConnect name associated with the OneFS System zoneUnfortunately, Zone-specific Role-Based Access Control (ZRBAC) is not fully supported by OneFS's RESTful Access to Namespace (RAN) service yet, which is required byisilon_create_directories.$iht_user, a OneFS System zone user with the following privileges:ISI_PRIV_LOGIN_PAPIISI_PRIV_AUTHISI_PRIV_HDFSISI_PRIV_IFS_BACKUP(only needed byisilon_create_directories)ISI_PRIV_IFS_RESTORE(only needed byisilon_create_directories)$zone, the name of the access zone on OneFS that will host HDFSThe System zone shouldNOTbe used for HDFS.$dist, the distribution of Hadoop that will be deployed with OneFS (e.g. CDH, HDP, etc.)$cluster_name, the name of the Hadoop clusterConnecting to OneFS via HTTPSOneFS ships with a self-signed SSL/TLS certificate by default, and such a certificate will not be verifiable by any well-known certificate authority. If you encounterCERTIFICATE_VERIFY_FAILEDerrors while using IHT, it may be because OneFS is still using the default certificate. To remedy the issue, consider encouraging your OneFS administrator to install a verifiable certificate instead. Alternatively, you may choose to skip certificate verification by using the--no-verifyoption, but do so at your own risk!Preparing OneFS for Hadoop DeploymentNote: This is not meant to be a complete guide to setting up Hadoop with OneFS. If you stumbled upon this page or have not otherwise consulted the appropriate install guide for your distribution, please do so athttps://community.emc.com/docs/DOC-61379.There are 2 tools in IHT that are meant to assist with the setup of OneFS as HDFS for a Hadoop cluster:isilon_create_users, which creates users and groups that must exist on all hosts in the Hadoop cluster, including OneFSisilon_create_directories, which sets the correct ownership and permissions on directories in HDFS on OneFSThese tools must be usedin ordersince a user/group must exist before it can own a directory.isilon_create_usersUsing the information from above, an invocation ofisilon_create_userscould look like this:isilon_create_users--dry\--onefs-user"$iht_user"\--zone"$zone"\--dist"$dist"\--append-cluster-name"$cluster_name"\"$onefs"Note:--drycauses the script to log without executing. Use it to ensure the script will do what you intend before actually doing it.If anything goes wrong (e.g. the script stopped because you forgot to give$iht_usertheISI_PRIV_HDFSprivilege), you can safely rerun with the same options. IHT should figure out that some of its job has been done already and work with what it finds.If a particular user/group already exists with a particular UID/GID, the ID it already has will be used.If a particular UID/GID is already in use by another user/group, IHT will try again with a different, higher ID.IHT mayNOTdetect previous runs that used different options.Generated Shell ScriptAfter runningisilon_create_users, you will find a new file in$PWDnamed like so:$unix_timestamp-$zone-$dist-$cluster_name.shThis script should be copied to and run on all the other hosts in the Hadoop cluster (excluding OneFS).
It will create the same users/groups with the same UIDs/GIDs and memberships as on OneFS using LSB utilities such asgroupadd,useradd, andusermod.isilon_create_directoriesUsing the information from above, an invocation ofisilon_create_directoriescould look like this:isilon_create_directories--dry\--onefs-user"$iht_user"\--zone"$zone"\--dist"$dist"\--append-cluster-name"$cluster_name"\"$onefs"Note:--drycauses the script to log without executing. Use it to ensure the script will do what you intend before actually doing it.If anything goes wrong (e.g. the script stopped because you forgot to runisilon_create_usersfirst), you can safely rerun with the same options. IHT should figure out that some of its job has been done already and work with what it finds.If a particular directory already exists but does not have the correct ownership or permissions, IHT will correct it.If a user/group has been deleted and re-created with a new UID/GID, IHT will adjust ownership accordingly.IHT mayNOTdetect previous runs that used different options.DevelopmentSee theContributing Guidelinesfor information on project development.