Package: aztk
Azure Distributed Data Engineering Toolkit (AZTK)Azure Distributed Data Engineering Toolkit (AZTK) is a python CLI application for provisioning on-demand Spark on Docker clusters in Azure. It's a cheap and easy way to get up and running with a Spark cluster, and a great tool for Spark users who want to experiment and start testing at scale.This toolkit is built on top of Azure Batch but does not require any Azure Batch knowledge to use.Notable FeaturesSpark cluster provision time of 5 minutes on averageSpark clusters run in Docker containersRun Spark on a GPU enabled clusterUsers can bring their own Docker imageAbility to use low-priority VMs for an 80% discountMixed Mode clusters that use both low-priority and dedicated VMsBuilt in support for Azure Blob Storage and Azure Data Lake connectionTailored pythonic experience with PySpark, Jupyter, and AnacondaTailored R experience with SparklyR, RStudio-Server, and TidyverseAbility to runspark submitdirectly from your local machine's CLISetupInstallaztkwith pip:pipinstallaztkInitialize the project in a directory. This will automatically create a.aztkfolder with config files in your working directory:aztksparkinitLogin or register for anAzure Account, navigate toAzure Cloud Shell, and run:wget-qhttps://raw.githubusercontent.com/Azure/aztk/v0.10.3/account_setup.sh-Oaccount_setup.sh&&chmod755account_setup.sh&&/bin/bashaccount_setup.shFollow the on screen prompts to create the necessary Azure resources and copy the output into your.aztk/secrets.yamlfile. For more information seeGetting Started Scripts.Quickstart GuideThe core experience of this package is centered around a few commands.# create your clusteraztksparkclustercreate
aztksparkclusteradd-user# monitor and manage your clustersaztksparkclusterget
aztksparkclusterlist
aztksparkclusterdelete# login and submit applications to your clusteraztksparkclusterssh
aztksparkclustersubmit1. Create and setup your clusterFirst, create your cluster:aztksparkclustercreate--idmy_cluster--size5--vm-sizestandard_d2_v2See our available VM sizeshere.The--vm-sizeargument must be the official SKU name which usually come in the form: "standard_d2_v2"You can createlow-priority VMsat an 80% discount by using--size-low-priinstead of--sizeBy default, AZTK runs Spark 2.2.0 on an Ubuntu16.04 Docker image. More infohereBy default, AZTK will create a user (with the usernamespark) for your clusterThe cluster id (--id) can only contain alphanumeric characters including hyphens and underscores, and cannot contain more than 64 characters.By default, you cannot create clusters of more than 20 cores in total. Visitthis pageto request a core quota increase.More information regarding using a cluster can be found in thecluster documentation2. Check on your cluster statusTo check your cluster status, use thegetcommand:aztksparkclusterget--idmy_cluster3. Submit a Spark jobWhen your cluster is ready, you can submit jobs from your local machine to run against the cluster. The output of the spark-submit will be streamed to your local console. Run this command from the cloned AZTK repo://submitajavaapplication
aztksparkclustersubmit\--idmy_cluster\--namemy_java_job\--classorg.apache.spark.examples.SparkPi\--executor-memory20G\path\to\examples.jar1000//submitapythonapplication
aztksparkclustersubmit\--idmy_cluster\--namemy_python_job\--executor-memory20G\path\to\pi.py1000Theaztk spark cluster submitcommand takes the same parameters as the standardspark-submitcommand, except instead of specifying--master, AZTK requires that you specify your cluster--idand a unique job--nameThe job name,--name, argument must be at least 3 characters longIt can only contain alphanumeric characters including hyphens but excluding underscoresIt cannot contain uppercase lettersEach job you submitmusthave a unique nameUse the--no-waitoption for your command to return immediatelyLearn more about the spark submit commandhere4. Log in and Interact with your Spark ClusterMost users will want to work interactively with their Spark clusters. With theaztk spark cluster sshcommand, you can SSH into the cluster's master node. This command also helps you port-forward your Spark Web UI and Spark Jobs UI to your local machine:aztksparkclusterssh--idmy_cluster--usersparkBy default, we port forward the Spark Web UI tolocalhost:8080, Spark Jobs UI tolocalhost:4040, and the Spark History Server tolocalhost:18080.You can configure these settings in the.aztk/ssh.yamlfile.NOTE: When working interactively, you may want to use tools like Jupyter or RStudio-Server. To do so, you need to setup your cluster with the appropriate docker image and plugin. SeePluginsfor more information.5. Manage and Monitor your Spark ClusterYou can also see your clusters from the CLI:aztk spark cluster listAnd get the state of any specified cluster:aztk spark cluster get --id <my_cluster_id>Finally, you can delete any specified cluster:aztk spark cluster delete --id <my_cluster_id>FAQsHow do I connect to Azure Storage (WASB)?I want to use a different version of SparkHow do I SSH into my Spark cluster's master node?How do I interact with my Spark cluster using a password instead of an SSH-key?How do I change my cluster default settings?How do I modify myspark-env.sh,spark-defaults.conforcore-site.xmlfiles?How do I use GPUs with AZTKI'm a python user and want to use PySpark, Jupyter, Anaconda packages, and have a Pythonic experience.I'm a R user and want to use SparklyR, RStudio, Tidyverse packages, and have an R experience.Next StepsYou can find more documentationhere