Package: aws-jupyter
This script launches a cluster on AWS EC2 instances and starts a Jupyter notebook on them.Please read themanualfor the details of all supported commands.QuickStartGuide.mdoffers a guided example to show whataws-jupytercan do.aws-jupyteris intended as a commandline tool, but you can integrate it to
a custom Python script.module-doc.mdprovides a brief how-to.InstallPleasure ensure you have Python 3.aws-jupytercan be install usingpip:pip install aws-jupyterAfter the installation, you can try it out using the example launch script:launch-aws-jupyterIt will create a cluster with 2 spot instances.In addition,
we create the EC2 instances using an AMI image that located in the regionus-west-2.
So please make sure that your local environment is set up to use that region
(you can runaws-jupyter configto verify the setting).After installation, please runaws-jupyter configto make sure the configuration is properly set.UpgradeIn case we change the default AWS region, please upgradeaws-jupyterin 3 steps:Upgrade the package:pip install --upgrade aws-jupyter.Create the new key pair in the new AWS region, and modify the "key_name" and "ssh_key" fields of
the credential file accordingly (read below).Switch to default AMI and region:aws-jupyter config --default-ami --default-region, and set the credential file
to the location of the new one when prompted.AWS CredentialThe scripts in this repository requires acredentials.ymlfile in following format:Arbitrary Name:access_key_id:your_aws_access_key_idsecret_access_key:your_aws_secret_access_keykey_name:your_ec2_key_pair_namessh_key:/path/to/the/ec2/key/pair/fileThe credential file in the Spark Notebook project can be directly used here.The credential file (or a soft link to it) should be located in the same folder where
you invoke these scripts (i.e. you should be able to see it usingls .command).
The credential file must alwaysstay privateand not be shared. Remember to addcredential.ymlto the.gitignorefile of your project so that this
file would not be pushed to GitHub.UsageRun any script in this directory with-hargument will print the help message of the script.Create a new clusteraws-jupyter createcreates a cluster on them3.xlargeinstance using an AMI based on Ubuntu.If the instance comes with attached SSD, it will be mounted to/mnt.Example:aws-jupytercreate-c2--nametestingCheck if a cluster is readyaws-jupyter checkchecks if a cluster is up and running. In addition, it also creates aneighbors.txtfile which contains the IP addresses of all the instances in the cluster.Exampleaws-jupytercheck--nametestingTerminate a clusteraws-jupyter terminateterminates a cluster by stopping and terminating all instances
in this cluster.Exampleaws-jupyterterminate--nametestingRun a script on a clusteraws-jupyter runruns a given script on all instances in the cluster.
It starts the script in the background, and redirect the stdout/stderr
into a file on the instances which can be checked later.
Thus it terminates does not necessarily mean the script has finshed executing on the cluster.
In addition, it only launches the script on all instances, but doesnotcheck if the script
executes without error.Exampleaws-jupyterrun--script./script-examples/hello-world.shSend a local directory to the remote instancesSend a local directory to all instances of a cluster.
It can be used to, for example, distributing the configuratoin files to all instances.Exampleaws-jupytersend-dir--local./configs/--remote~/remote-configsRetrieve files from all instances in a clusterRetrieve files from the same location on all instances of a cluster.
It can be used to collect the output of the program from the workers.
A local directory for saving the downloaded files should be provided to this script.
This script will create a separate sub-directory for each worker and download its files
to this sub-directory.Examplemkdir_result
aws-jupyterretrieve--remote/tmp/std*--local./_result/Install a package on all instancesYou can install any missing packagesafter the instances are ready(i.e.aws-jupyter checkshows Jupyter notebook URL) by following these step:create a script on your local computer, which install the required package. For example, assume we want to installpandas,$echo"pip install pandas">install-pandas.shrun the script on all instancesaws-jupyterrun-sinstall-pandas.sh--outputThe--outputargument ensures that the script will run in foreground, so that you can check if the installation succeed.