Package: sagecreator
SageCreatorSageCreator is a package meant to simplify cluster setup for Machine Learning in AWS.It does all the heavy lifting to get cluster up and running in a matter of minutes on any of the AWS instance type(s).It usesspot instancesby default which can significantly reduce total cost of running the cluster.If spot instances are not available or the specified spot instance price is too low, it falls back toon-demandinstances.You can accessJupyter notebookthat can run your code against provisioned server(s). SeeJupyter accessfor more info.InstallationInstall and update usingpip:$ pip install sagecreatorPython 3 is required and it is highly recommended to install and run the package invirtualenv.
Supported in Python >= 3.5.0.PrerequisitesAWS AccountTo provision the cluster you need anAWS Accountand an IAM user with:Access Key IDSecret Access KeyUser should either be inAdministratorsgroup as described inIAM usertutorial, or create acustom IAM policyExecutionAfter the installation, configure the tool by specifying configuration parameters:$ sage configure
Access key id: <AWS Access Key ID>
Secret access key: <AWS Secret Access Key>
Company: <Name of your organization>
Owner: <Name of your team>
Key pair name: <Name of the key pair> (Optional - if NOT provided it will be created with a new private key)
Private key file: <Absolute path to private key file> (required only if Key pair name was provided)Company,Owner,Serviceare required - those are used as tags for each instance in the cluster.Key pair name,Private key fileare optional - if provided, given ‘key pair name’ / ‘private key file’ will be used to provision the cluster.Provision the cluster.Provision step can take up to 20 minutes depending on the network connection, cluster size, and instance type.$ sage provision
Service: <Name of your service>
Instance type [t3.small]: <Instance type> (Optional, defaults to t3.small)
Spot instance price [0.1]: <Spot instance price> (Optional, defaults to $0.1 per instance)
Cluster size [1]: <Cluster size> (Optional, defaults to 1 node)Important:The tool provides NO guarantee that the instance(s) will be provisioned at specifiedSpot instance price.If specified price is lower than the current AWS spot instance price thenOn-demandinstance(s) will be provisioned instead.Thus, it is up to the user to ensure that specified price is high enough for the request to be fulfilled.Display path of the cluster configuration file.Though not necessary it is possible to manually edit that file with customizations prior to runningprovisionstep.$ sage pwdTerminate cluster. This operation terminates all cluster nodes matching tags tuple ofCompany,Owner,Service.$ sage terminate
Service: <Name of your service to terminate>Jupyter accessOnce provisioning step is done and the cluster is up you can access jupyter notebook in your browser athttp://localhost:9000.We have provided a sample notebook to execute. It trains the model on Fashion MNIST dataset using CNN in Keras.Under the hoodThe logic that orchestrates the cluster is written inAnsibleCustom IAM policyAlternatively, instead of assigning user toAdministratorsgroup which has access to all AWS services (as described inIAM user), you can create separate Group namedProvisionerswith more restrictive policy:{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "ec2:*",
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Action": "rds:*",
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Action": "route53:*",
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}Then assign the user to theProvisionersgroup which has access to a subset of AWS services that are sufficient to orchestrate the cluster.SSH accessIfKey pair name/Private key filewere NOT provided when configuring the cluster then default key pair is created and a new private key is stored locally.In order tosshinto the servers pointsshto the correct (private key) file:$ ssh -i <path to private key file> ubuntu@<host>wherepath to private key fileis../venv/lib/python3.X/site-packages/sagebase/.ssh/pkey.pem