Package: cloudless
CloudlessCloudless makes it easier to interact with cloud resources by doing most of the
work that a human doesn't need to care about for you, and by being transparent
about what it's doing.HomepageDocumentationGithubInstallationThis project depends onPython
3.6.0or greater.  To
install, run:pip3installcloudlessQuick StartThese examples show how to quickly create a simple service that is accessible on
port 80 in Google Compute Engine and in Amazon Web Services.  Run any command
with--helpfor more usage.Google Compute Engine CredentialsTo set up the Google Compute Engine client, you must first create your Google
Compute Engine account and navigate tohttps://console.cloud.google.com/iam-admin/serviceaccounts.
There you can select your project and create a service account.  Remember the
service account email and create a key for this service account.  Download and
save this key on your local machine, remembering the path.You will also need the project ID (not the project name).  Go tohttps://console.cloud.google.com/iam-admin/settings/projectand select your project to find your project ID.  When you think you have
everything, run:cldlsinit--providergce
cldlsnetworklistThis will set the "default" profile to use the "gce" backend.  You can change
the profile by passing--profileto cloudless or setting theCLOUDLESS_PROFILEenvironment variable.  SeeProfilesfor more information.Amazon Web Services CredentialsTo set up the Amazon Web Services client, follow the steps athttps://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.htmlto configure the aws cli.  Currently, cloudless only uses the "default" aws
profile that is configured in this way.  After you set up a default profile that
works with the AWS cli, everything in cloudless should work.  When you think you
have this working, run:cldls init --provider aws
cldls network listThis will set the "default" profile to use the "aws" backend.  You can change
the profile by passing--profileto cloudless or setting theCLOUDLESS_PROFILEenvironment variable.  SeeProfilesfor more information.Simple ServiceThe next steps are identical regardless of which cloud provider you're using.
They do assume you have an image named "cloudless-example-base-image-v0" with
Ubuntu 16.04 installed on it in your provider.  See theImage
Buildersection for how to build an image like this using
Cloudless.Once you have the prerequisites, you can create your service with:cldlsnetworkcreatemynetexamples/network/blueprint.yml
cldlsservicecreatemynetmyserviceexamples/apache/blueprint.yml
cldlspathsallow_network_blockmynetmyservice0.0.0.0/080cldlsservicegetmynetmyservice# Navigate to the "public_ip" of each instance in a browser to see the service.Command Line AutocompleteSince this project usesclick,
autocomplete is built in.  Just followhttps://click.palletsprojects.com/en/5.x/bashcomplete/if you use bash orhttps://github.com/click-contrib/click-completionfor other shells.For example, for bash puth this in your .bashrc:eval"$(_CLDLS_COMPLETE=sourcecldls)"ProfilesBoth the API and the command line support using profiles that are created withcldls init.  The
order of priority for loading profiles is:Explicitly set via theprofileargument tocloudless.Clientin the python api, or via the--profileoption to thecldlscommand line.Set in theCLOUDLESS_PROFILEenvironment variable."default"Client Setup In Python APIIn the Python API, you must create a client object to connect to the cloud
platform that you'll be working with.  The client handles authentication with
the cloud provider, so you must pass it the name of the provider and the
authentication credentials.If you are trying this project for the first time, it's recommended that you use
the "mock-aws" client.Google Compute Engine ClientTo use the Google Compute Engine client, you must create a service account and
download the credentials locally.  Because this provider is implemented usingApache Libcloud, you can refer to theGoogle
Compute Engine Driver
Setupdocumentation in that project for more details.When you have the credentials, you can do something like this, preferably in a
dotfile you don't commit to version control.  Note the credentials file is in
JSON format:exportCLOUDLESS_GCE_USER_ID="sverch-cloudless@cloudless-000000.iam.gserviceaccount.com"exportCLOUDLESS_GCE_CREDENTIALS_PATH="/home/sverch/.gce/credentials.json"exportCLOUDLESS_GCE_PROJECT_NAME="cloudless-000000"Then, you can run these commands in a python shell to create a GCE client:importcloudlessimportosclient=cloudless.Client("gce",credentials={"user_id":os.environ['CLOUDLESS_GCE_USER_ID'],"key":os.environ['CLOUDLESS_GCE_CREDENTIALS_PATH'],"project":os.environ['CLOUDLESS_GCE_PROJECT_NAME']})If you want to avoid having to pass all this configuration explicitly to the client object, you can
use aCloudless Profile.Amazon Web Services ClientCurrently no credentials can be passed in as arguments for the AWS provider
(they are ignored).  However this provider is implemented withBoto, which looks in many other places
for the credentials, so you can configure them in other ways.  See theboto3
credential setup
documentationfor more details.Once you have set up your credentials, you can run the following to create an AWS client that uses
the "default" aws profile (if you pass an empty credentials object, this cloudless profile will use
whatever theAWS_PROFILEenvironment variable is set to, which might be confusing):importcloudlessclient=cloudless.Client("aws",credentials={"profile":"default"})If you want to avoid having to pass all this configuration explicitly to the client object, you can
use aCloudless Profile.Mock Amazon Web Services ClientThe Mock AWS client is for demonstration and testing.  Since it is all running
locally, you don't need any credentials.  Simply run:importcloudlessclient=cloudless.Client("mock-aws",credentials={})ArchitectureThere are only three objects in Cloudless: A Network, a Service, and a Path.  This
is an example that shows a Networkdev, apublic_load_balancerService, aninternal_serviceService, a Path from the internet topublic_load_balanceron port 443, and a Path frompublic_load_balancertointernal_serviceon
port 80.  See thevisualizationsection for how to generate
this graph.NetworkA Network is the top level container for everything else.  To create a new
network, run:dev_network=client.network.create("dev")This will return the "Network" object that describes the network that was
created.  You can retrieve an existing network or list all existing networks by
running:dev_network=client.network.get("dev")all_networks=client.network.list()Finally, to destroy a network:client.network.destroy(dev_network)Create should use sane defaults, but if you need to do something special seedocs/network-configuration.md.Inipython, you can run<object>?toget help on any
object,
for exampleclient.network.create?.ServiceA Service a logical group of instances and whatever resources are needed to
support them (subnetworks, firewalls, etc.).To create a Service, you must first define a configuration file called a
"blueprint" that specifies how the service should be configured.  This is an
example of what a Service blueprint might look like:---network:subnetwork_max_instance_count:768placement:availability_zones:3instance:public_ip:Truememory:4GBcpus:1gpu:falsedisks:-size:8GBtype:standarddevice_name:/dev/sda1image:name:"ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-*"initialization:-path:"haproxy-cloud-config.yml"vars:PrivateIps:required:trueThe "network" section tells Cloudless to create subnetworks for this service big
enough for 768 instances.The "placement" section tells Cloudless to ensure instances in this service are
provisioned across three availaibility zones (which most cloud providers
guarantee are meaningfully isolated from each other for resilience).The "instance" section describes the resource reqirements of each instance.
Cloudless will automatically choose a instance type that meets these requirements.The "image" section represents the name of the image you want your instances to
have.  In this case, we are using an image name only found in AWS by default, so
this example will only work there.  Seeexamples/apachefor a blueprint that
works cross cloud because it uses a custom image.The "initialization" section describes startup scripts that you want to run when
the instance boots.  You may also pass in variables, which will get passed to
the given file asjinja2template arguments.  This is
a good place to specify environment specific configuration, so your base image
can stay the same across environments.Once you have the blueprint, the example below shows how you could use it.
These examples create a group of private instances and then create some HAProxy
instances in front of those instances to balance load.  Note that many commands
takedev_networkas the first argument.  That's the same network object
returned by the network commands shown above.  These assume you have created the
base image inexamples/base-imageon the provider you are using.internal_service=client.service.create(dev_network,"private",blueprint="examples/nginx/blueprint.yml")private_ips=[instance.private_ipforinstanceinclient.service.get_instances(internal_service)]load_balancer_service=client.service.create(dev_network,"public",blueprint="examples/haproxy/blueprint.yml",template_vars={"PrivateIps":private_ips})internal_service=client.service.get(dev_network,"public")load_balancer_serviceclient.service.get(dev_network,"private")client.service.list()client.service.destroy(internal_service)client.service.destroy(load_balancer_service)PathThe Path is how you tell Cloudless that two services should be able to communicate.
No blueprint is needed for this, but you need to have the service objects you
created earlier.  This example adds a path from the load balancer to the
internal service on port 80 and makes the load balancer internet accessible on
port 443:fromcloudless.types.networkingimportCidrBlockinternet=CidrBlock("0.0.0.0/0")client.paths.add(load_balancer_service,internal_service,80)client.paths.add(internet,load_balancer_service,443)You can check whether things have access to other things or print out all paths
with the following functions:client.paths.has_access(load_balancer_service,internal_service,80)client.paths.internet_accessible(load_balancer_service,443)client.paths.internet_accessible(internal_service,443)client.paths.list()print(client.graph())VisualizationGet a summary in the form of a graphviz compatible dot file by running:client.graph()To generate the vizualizations, run:cdui&&envPROVIDER=<provider>bashgraph.shAnd openui/graph.htmlin a browser.  Note this won't work for themock-awsprovider since it will be running in a different process.Image BuilderThis project provides a cross cloud image builder that depends on the core
cloudless APIs.  this means that for the most part it is completely cloud
independent, mod differences in the image that you start with (so it's
completely independent if you're building your own custom image from scratch).For this example, we use the Ubuntu image provided by the cloud provider, so we
have different blueprints for AWS and GCE (because the standard Ubuntu images
they provide have different names).First, to deploy a service running a single instance:$cldls--profilegceimage-builddeployexamples/base-image/gce_image_build_configuration.ymlNext, to run the configure script.  This is a shellscript that cloudless will
pass the login credentials to as arguments, and where you can run your
configuration as code scripts:$cldls--profilegceimage-buildconfigureexamples/base-image/gce_image_build_configuration.ymlNext, to run the check script.  This is another shellscript that cloudless will
pass the login credentials to as arguments, and where you can run your
validation to make sure the configuration step worked as expected:$cldls--profilegceimage-buildcheckexamples/base-image/gce_image_build_configuration.ymlFinally, when you have your scripts working as you want them to, run a cleanup
in preparation for a full build.  Saving images without a full build is not
supported to discourage modifications that are made on the machine and not
recorded in scripts anywhere making it into the image:$cldls--profilegceimage-buildcleanupexamples/base-image/gce_image_build_configuration.ymlNow, run the full build end to end, and you have your new image!$cldls--profilegceimage-buildrunexamples/base-image/gce_image_build_configuration.ymlWe can list the image with:$cldls--profilegceimagelist
ImageName:cloudless-example-base-image-v0
ImageId:ami-0d7366265fcccbe46
ImageCreatedAt:2018-09-20T16:51:03.000ZGet it by name with:$cldls--profilegceimagegetcloudless-example-base-image-v0
ImageName:cloudless-example-base-image-v0
ImageId:ami-0d7366265fcccbe46
ImageCreatedAt:2018-09-20T16:51:03.000ZAnd finally, delete the image.  You might want to wait on this step because theService Testerstep below uses this image:$cldls--profilegceimagedeletecloudless-example-base-image-v0
Deletedimage:cloudless-example-base-image-v0Seeexamples/base-imagefor examples of how to create a
cross cloud base image using this framework.Service TesterThis project also provides a framework to help test that blueprint files work as
expected.  The framework will create, verify, and clean up the service under
test.  It also spins up all dependent services so you can test services "in
context".  It's sort of a hybrid between a unit test and an integration test.These examples assume the profile you have configured has an image that was
built using theImage Builderstep above.  If you've followed
those steps, these instructions are completely identical regardless of whether
you're using AWS or GCE.First, to create the service:$cldlsservice-testdeployexamples/apache/service_test_configuration.yml
Creationcomplete!
Tologin,run:
ssh-iexamples/apache/id_rsa_testcloudless_service_test@35.237.12.140
$ssh-iexamples/apache/id_rsa_testcloudless_service_test@35.237.12.140
...
...

UbuntucomeswithABSOLUTELYNOWARRANTY,totheextentpermittedby
applicablelaw.

cloudless@test-network-jmeiwknbsg-test-service-hswonxmeda-0:~$This will create a temporary network to sandbox the test.  Now, to verify that
the service is behaving as expected:$cldlsservice-testcheckexamples/apache/service_test_configuration.yml
INFO:cloudless.providers.gce:Discoveringsubnetworktest-network-jmeiwknbsg,test-service-hswonxmeda
INFO:cloudless.util:Attemptnumber:1INFO:cloudless.util:Checksuccessful!
Checkcomplete!
Tologin,run:
ssh-iexamples/apache/id_rsa_testcloudless@35.237.12.140Finally, to clean up everything:$cldlsservice-testcleanupexamples/apache/service_test_configuration.ymlIf you want to run all the previous steps all together, you can run:$cldlsservice-testrunexamples/apache/service_test_configuration.ymlSeeexamplesfor examples of how to set up a blueprint to be
testable with this framework.TestingTo run the local tests run:pipenvinstall--dev
toxTo run tests against GCE and AWS, run:tox-egce
tox-eawsThese will use thegce-cloudless-testandaws-cloudless-testcloudless
profiles respectively.  SeeProfilesfor more information.