Package: hadoop-test-cluster
A dockerized setup for testing code on a hadoop cluster.Installationhadoop-test-clusteris available on PyPI:$pipinstallhadoop-test-clusterYou can also install from source on github:$pipinstallgit+https://github.com/jcrist/hadoop-test-cluster.gitOverviewFor testing purposes, infrastructure for setting up a mini hadoop cluster using
docker is provided here. Two base images are provided:cdh5: provides a CDH5 installation of Hadoop 2.6cdh6: provides a CDH6 installation of Hadoop 3.0Both images can be run with 2 different configurations:simple: usessimpleauthentication (unix user permissions)kerberosuseskerberosfor authenticationEach cluster has three containers:Onemasternode running thehdfs-namenodeandyarn-resourcemanager, as well as the kerberos daemons.Oneworkernode running thehdfs-datanodeandyarn-nodemanagerOneedgenode for interacting with the clusterOne user account has also been created for testing purposes:Login:testuserPassword:testpassFor thekerberossetup, a keytab for this user has been put at/home/testuser/testuser.keytab, so you can kinit easily likekinit-kt/home/testuser/testuser.keytab testuserAn admin kerberos principal has also been created for use withkadmin:Login:admin/adminPassword:adminpassPortsThe full address is dependent on the IP address of your docker-machine driver,
which can be found at:$docker-machineinspect--format{{.Driver.IPAddress}})NameNode RPC: 9000NameNode Webui: 50070ResourceManager Webui: 8088Kerberos KDC: 88Kerberos Kadmin: 749DataNode Webui: 50075NodeManager Webui: 8042Thehtclustercommandline toolTo work with either cluster, please use thehtclustertool. This is a thin
wrapper arounddocker-compose, with utilities for quickly doing most common
actions.$htcluster--helpusage: htcluster [--help] [--version] command ...

Manage hadoop test clusters

positional arguments:
command
    startup   Start up a hadoop cluster.
    login     Login to a node in the cluster.
    exec      Execute a command on the node as a user
    shutdown  Shutdown the cluster and remove the containers.
    compose   Forward commands to docker-compose
    kerbenv   Output environment variables to setup kerberos locally. Intended
              use is to eval the output in bash: eval $(htcluster kerbenv)

optional arguments:
--help, -h  Show this help message then exit
--version   Show version then exitStarting a clusterStart a CDH5 cluster with simple authentication:$htclusterstartup--imagecdh5--configsimpleStart a CDH6 cluster with kerberos authentication$htclusterstartup--imagecdh6--configkerberosStarting a cluster, mounting the current directory to ~/workdir$htclusterstartup--imagecdh5--mount.:workdirLogin to the edge node$htclusterloginRun a commmand as the user on the edge node$htclusterexec--myscript.shsomeotherargsShutdown the cluster$htclustershutdownAuthenticating with Kerberos from outside DockerIn the kerberized cluster, the webui’s are secured by kerberos, and so won’t be
accessible from your browser unless you configure kerberos properly. This is
doable, but takes a few steps:Kerberos/SPNEGO requires that the requested url matches the hosts domain.
The easiest way to do this is to modify your/etc/hostsand add a line formaster.example.com:#Addalineto/etc/hostspointingmaster.example.comtoyourdocker-machine#driveripaddress.#Notethatyouprobablyneedtorunthiscommandasasuperuser.$echo"$(docker-machineinspect--format{{.Driver.IPAddress}})master.example.com">>/etc/hostsYou must havekinitinstalled locally. You may already have it, otherwise
it’s available through most package managers.You need to tell kerberos where thekrb5.confis for this domain. This is
done with an environment variable. To make this easy,htclusterhas a
command to do this:$eval$(htclusterkerbenv)At this point you should be able to kinit as testuser:$kinittestuser@EXAMPLE.COMTo access kerberos secured pages in your browser you’ll need to do a bit of
(simple) configuration. See [this documentation from
Cloudera](https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cdh_sg_browser_access_kerberos_protected_url.html)
for information on what’s needed for your browser.Since environment variables are only available for processes started in the
environment, you have three options here:Restart your browser from the shell in which you added the environment
variablesManually get a ticket for theHTTP/master.example.comprincipal. Note
that this will delete your other tickets, but works fine if you just want
to see the webpage$kinit-SHTTP/master.example.comtestuserUsecurlto authenticate the first time, at which point you’ll already
have the proper tickets in your cache, and the browser authentication will
just work. Note that your version of curl must support the GSS-API.$curl-V# Check your version of curl supports GSS-APIcurl 7.59.0 (x86_64-apple-darwin17.2.0) libcurl/7.59.0 SecureTransport zlib/1.2.11
Release-Date: 2018-03-14
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets$curl--negotiate-u:http://master.example.com:50070# get a HTTP ticket for master.example.comAfter doing one of these, you should be able to access any of the pages from
your browser.