Package: cloudlift
CloudliftCloudlift is built by Simpl developers to make it easier to launch dockerized
services in AWS ECS.Cloudlift is a command-line tool for dockerized services to be deployed in AWS
ECS. It's very simple to use. That's possible because this is heavily
opinionated. Under the hood, it is a wrapper to AWS cloudformation templates. On
creating/updating a service or a cluster this creates/updates a cloudformation
in AWS.Demo videosCreate EnvironmentCreate ServiceDeploy Service with image buildDeploy ServiceInstalling cloudlift1. Pre-requisitespipcurlhttps://bootstrap.pypa.io/get-pip.py-oget-pip.py|pythonget-pip.py2. Install cloudliftpipinstallcloudlift3. Configure AWSawsconfigureEnter the AWS Access Key ID, AWS Secret Access Key. You can find instructions
here on how to get Access Key ID and Secret Access Key here athttp://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.htmlUsing AWS ProfilesIf you are usingAWS profiles, set the desired profile name in the environment before invoking Cloudlift.AWS_DEFAULT_PROFILE=<profilename>cloudlift<command>ORexportAWS_DEFAULT_PROFILE=<profilename>
cloudlift<command>
cloudlift<command>UsageCreate a new environmentCreate a new environment for services to be deployed. Cloudlift creates a new
VPC for the given CIDR and sets up the required networking infrastructure for
services to run in ECS.cloudliftcreate_environment-e<environment-name>This starts a prompt for required details to create an environment, which
includes -AWS region for the environmentVPC CIDRNAT Elastic IP allocation ID2 Public Subnet CIDRs2 Private Subnet CIDRsMinimum instances for clusterMaximum instances for clusterSSM parameter path of Custom AMI IDSSH key nameSNS ARN for notificationsAWS ACM ARN for SSL certificateOnce the configuration is saved, this is opened in the defaultVISUALeditor.
Here configurations can be changed if required.Update an environmentcloudliftupdate_environment-e<environment-name>This opens the environment configuration in theVISUALeditor. Update this to
make changes to the environment.Create a new serviceObject StructureThe configuration object is structured as follows:{"notifications_arn":"string","services":{"Test123":{"command":"string/null","custom_metrics":{"metrics_port":"string","metrics_path":"string"},"http_interface":{"container_port":"number","internal":"boolean","restrict_access_to":["string","string"]},"volume":{"efs_id":"string","efs_directory_path":"string","container_path":"string"},"memory_reservation":"number","logging":"string/null"}}}Required FieldsThe following fields are required in the configuration object:notifications_arn: A string representing the Amazon Resource Name (ARN) of the SNS topic to which notifications will be sent.services: An object representing the services to be configured. The keys of this object are the names of the services, and the values are objects containing configuration information for each service.Service ConfigurationEach service object must contain the following fields:command: A string or null value representing the command to be run in the Docker container. If this field is null, the command specified in the Dockerfile will be used.memory_reservation: A number representing the soft memory limit for the container. The hard limit will automatically be set to 1.5 times the soft limit.In addition, a service object may contain any of the following optional fields:custom_metrics: An object containing configuration information for exporting custom metrics to a Prometheus server. This field is only used if custom metrics are required.NOTE:If you use custom metrics, Your ECS container Network mode will beawsvpc.⚠ WARNING:If you are adding custom metrics to your existing service, there will be a downtime.metrics_port: A string representing the port number on which custom metrics are exported.metrics_path: A string representing the path on which custom metrics are exported.http_interface: An object containing configuration information for setting up an Application Load Balancer (ALB) for the service. This field is only used if an ALB is required.container_port: A number representing the port number on which the service is running inside the container.internal: A boolean value indicating whether the ALB should be internal. If set to false, the ALB will be public.restrict_access_to: An array of strings representing the IP addresses that should be allowed to access the ALB.volume: An object containing configuration information for mounting an Amazon Elastic File System (EFS) volume to the service. This field is only used if an EFS volume is required.efs_id: A string representing the ID of the EFS volume to be mounted.efs_directory_path: A string representing the directory path on the EFS volume to be mounted.container_path: A string representing the mount path inside the container.logging: A string or null value representing the log driver to be used. Valid options are "fluentd", "awslogs", or null. If this field is null, the default log driver (CloudWatch Logs) will be used.1. Upload configuration to Parameter StoreDuring create_service and deploymentcloudliftpulls the config from AWS
Parameter Store to apply it on the task definition. Configurations are stored in
path with the convention/<environment>/<service>/<key>cloudliftedit_config-e<environment-name>NOTE: This isnotrequired for every deployment. It's required only when
config needs to be changed.2. Create serviceIn the repository for the application, run -cloudliftcreate_service-e<environment-name>This opens theVISUALeditor with default config similar to -{"notifications_arn":"<SNS Topic ARN>","services":{"Test123":{"command":null,"http_interface":{"container_port":80,"internal":false,"restrict_access_to":["0.0.0.0/0"]},"memory_reservation":100}}}3. Deploy serviceThis command build the image (only if the version is unavailable in ECR), pushes to ECR and updates the ECS
service task definition. It supports--build-argargument ofdocker buildcommand as well to pass
custom build time argumentscloudliftdeploy_service-e<environment-name>For example, you can pass your SSH key as a build argument to docker buildcloudliftdeploy_service--build-argSSH_KEY"\"`cat ~/.ssh/id_rsa`\""-e<environment-name>This example is bit comprehensive to showit can execute shell commands with "`".It's wrapped with double quotes to avoid line-breaks in SSH keys breaking the command.4. Starting shell on container instance for serviceYou can start a shell on a container instance which is running a task for given
application using thestart_sessioncommand. One pre-requisite for this is
installing the session manager plugin forawscli. To install session manager
plugin follow theguidecloudliftstart_session-e<environment-name>MFA code can be passed as parameter--mfaor you will be prompted to enter
the MFA code.Example1. Service configuration:{"notifications_arn":"arn:aws:sns:us-east-1:123456789012:MyTopic","services":{"Test123":{"command":null,"custom_metrics":{"metrics_port":"8080","metrics_path":"/metrics"},"http_interface":{"container_port":3000,"internal":false,"restrict_access_to":["192.0.2.0/24","198.51.100.0/24"]},"volume":{"efs_id":"fs-0123456789abcdef","efs_directory_path":"/mydata","container_path":"/data"},"memory_reservation":256,"logging":"fluentd"}}}In this example, we are configuring a service named Test123. The service has the following configuration:command: The command to be run in the Docker container is not specified, so the command specified in the Dockerfile will be used.custom_metrics: Custom metrics will be exported to a Prometheus server running on port 8080, with the metrics available at the path "/metrics".http_interface: An Application Load Balancer (ALB) will be set up for the service on port 3000. The ALB will be public and will allow access only from the IP addresses in the restrict_access_to array.volume: An Amazon Elastic File System (EFS) volume with ID fs-0123456789abcdef will be mounted to the service. The volume will be mounted at the directory path "/mydata" on the EFS volume, and at the path "/data" inside the container.memory_reservation: The soft memory limit for the container is set to 256 MB, so the hard limit will be automatically set to 384 MB.logging: Logs will be sent to a Fluentd log driver.2. Service configuration with custom metrics:{"notifications_arn":"<SNS Topic ARN>","services":{"Test123":{"command":null,"custom_metrics":{"metrics_port":"8005","metrics_path":"/metrics"},"http_interface":{"container_port":80,"internal":false,"restrict_access_to":["0.0.0.0/0"]},"memory_reservation":100}}}3. Service configuration with volume mount:{"notifications_arn":"<SNS Topic ARN>","services":{"Test123":{"command":null,"volume":{"efs_id":"fs-XXXXXXX","efs_directory_path":"/","container_path":"/"},"http_interface":{"container_port":80,"internal":false,"restrict_access_to":["0.0.0.0/0"]},"memory_reservation":100}}}4. Service configuration with http interface only:{"notifications_arn":"<SNS Topic ARN>","services":{"Test123":{"command":null,"http_interface":{"container_port":80,"internal":false,"restrict_access_to":["0.0.0.0/0"]},"memory_reservation":100}}}5. Service configuration with http interface without AWS CW logging.NOTE:Do not uselogging: nullin production. Once container  gets deleted all logs will be lost. Logging configuration should be one of the following:awslog,fluentd,null{"notifications_arn":"<SNS Topic ARN>","services":{"Test123":{"command":null,"http_interface":{"container_port":80,"internal":false,"restrict_access_to":["0.0.0.0/0"]},"memory_reservation":100,"logging":null}}}Contributing to cloudliftSetupUse the latest git mastergitclonegit@github.com:GetSimpl/cloudlift.gitcdcloudlift
./install-cloudlift.shTo ensure the tests use the development version and not the installed version run (referhere)pip install -e .TestsFirst level of tests have been added to assert cloudformation template generated
vs expected one.py.testtest/deployment/To run high level integration testspytest-stest/test_cloudlift.pyThis tests expects to have an access to AWS console.
Since there's no extensive test coverage, it's better to manually test the
impacted areas whenever there's a code change.