Package: haldane
a friendly http interface to the ec2 apiRequirementsVagrantVMWare Fusion1 gigabytes of free ramPython 2.7.8 or below due to a bug in 2.7.9 SSLInstallationOS X Users are assumed to be running homebrew. Please set that up before continuing.
Our makefile should install everything necessary to run this service within a Ubuntu 14.04 or Mac OS X 10.9+ environment.On a VMOnce you have vagrant and virtualbox installed, you can bring up a vm with the service installed:cdpath/to/haldanevagrantupBy default, this will usevmware_fusionas the vagrant provider, though using you can also usevirtualbox:vagrantup--providervirtualboxManuallyYou will need to setup a few system-level dependencies first. You can do this using the following make target:# install system-level dependencies first
# may ask for sudo permissions to install certain packagesmakerequirements# now create a python virtual environment with all the required packages
# will ask for sudo permissions to install virtualenv if it is not available
# the path is set to ``.virtualenv`` by default, though we respect
# the following environment variables:
# - ``WORKON_HOME``
# - ``VIRTUALENV_PATH``makevenv# activate the virtualenv:eval$(makevenv-activate)# to deactivate the virtualenv, run the following command:deactivate# for more actions you can perform, just run the ``make`` or ``make help`` commandsRunningYou can ssh onto the box and run the webservice:vagrantsshsource.env&&makeserverThe webservice will now be running and exposed to the host machine athttp://localhost:5000> Note that you may wish to change the configuration in use. You can do so by modifying the .env.test file with your configuration.ConfigurationAll configuration is set via environment variables. The following environment variables are available for use:ALLOWED_IPS: (Default: None) A comma-separated list of ip addresses that can basic authentication.ALTERNATIVE_AUTOSCALE_TAG_NAME: (Default: None) A tag that can be used as an alternative to the AWS group name to categorize instances.AWS_API_VERSION: (Default:2016-09-15) The default api version to use when retrieving instance type.AWS_ACCESS_KEY_ID: (Default: None) An AWS access key idAWS_REGIONS: (Default: us-east-1) A comma-separated list of regions to query for.AWS_SECRET_ACCESS_KEY: (Default: None) An AWS secret access keyBASIC_AUTH: (Default: None) A list of basic auth user/password combinations. The format for each isusername:password.BOOLEAN_AWS_TAG_ATTRIBUTES: (Default: None) A comma-separated list of instance tags that will be pulled out as top-level instance attributes set and converted into booleans.BUGSNAG_API_KEY: (Default: None) An api key for reporting errors to bugsnag.CACHE_EXPIRATION: (Default:180) Time in seconds until a cached AWS api retrieval expires.CACHE_SIZE: (Default:1024) Max number of items to cache in the LRU cache. Can be safely set to 2.DEBUG: (Default:0) Whether to turn on debug mode or not.LISTEN_INTERFACE: (Default:0.0.0.0) The interface which the server will bind to.PORT: (Default:5000) Server port.SENTRY_DSN: (Default: None) An DSN for reporting errors to sentry.TOP_LEVEL_AWS_TAG_ATTRIBUTES: (Default: None) A comma-separated list of instance tags that will be pulled out as top-level instance attributes set.The AWS policy is fairly small, and aniam-profile.jsonis provided in this repository in the case that you wish to lock down permissions to only those necessary.Endpoints/: Healthcheck/_status: Healthcheck/amis?q=<query>&limit=<limit>: List all amis owned by the user specified by the AWS credentials.
-format(optional): If set tolist, turns ami attributes from an object indexed by the name key to a list of those objects. Can also be set tocsv. Defaults todict.
-id(optional): An image id to filter by (eg.ami-21e750d9)
-query(optional): Substring to search ami names by before returning the resultset
-region(optional): Filter to a specific region/instance-types/<api-version>: List all instance types available for a specific api version (version is optional)./instances/<region>?q=<query>&limit=<limit>&status=<status>&group=<group>: List all nodes
-format(optional): If set tolist, turns node attributes from an object indexed by the name key to a list of those objects. Can also be set tocsv. Defaults todict.
-limit(optional): An integer to limit the resultset by
-query(optional): Substring to search thenamefield by before returning the resultset/instances/group/<group>?region=<region>&query=<query>&status=<status>: List all nodes grouped by autoscale group
-format(optional): If set tolist, turns node attributes from an object indexed by the name key to a list of those objects.
-query(optional): Substring to search node names by before returning the resultset/rds-instances/<region>?q=<query>&limit=<limit>&status=<status>: List all nodes
-format(optional): If set tolist, turns node attributes from an object indexed by the name key to a list of those objects. Can also be set tocsv. Defaults todict.
-limit(optional): An integer to limit the resultset by
-query(optional): Substring to search thenamefield by before returning the resultsetFiltersField values that areexactlyone of the following strings are transformed into their language “equivalents”:noneniltruefalseThe values are lowercased before the check is performed.The following attributes filters are globally available:availability_zone(optional): An availability zone to filter by (eg.us-east-1a)id(optional): An instance id to filter by (eg.i-21e750d9)name(optional): A name to filter by (eg.graphite-ec2-01)region(optional): Filter to a specific region (eg.us-east-1)status(optional): Filter to specific node status (eg.terminated)The following attribute filters are available for the/instancesand/instances/groupendpoints:elastic_ip(optional): Whether or not to filter to just instances with an elastic_ipgroup(optional): An autoscale group name to filter byimage_id(optional): An image id to filter by (eg.ami-123abc4d)image_name(optional): An image name to filter by (eg.BaseAMI)instance_type(optional): Filter to a specific instance type (eg.t2.large)instance_class(optional): Filter to a specific instance type (eg.t2)instance_profile_id(optional): Filter to a specific instance profile id (eg.O34RQ3IUIO3FOUI3F)instance_profile_name(optional): Filter to a specific instance profile name (eg.Production-Api)ip_address(optional): Filter to a specific private ip address (eg.54.10.2.20)private_ip_address(optional): Filter to a specific private ip address (eg.10.10.2.20)vpc_id(optional): Filter to specific vpc (eg.vpc-8675309)The following attribute filters are available for the/rds-instancesendpoint:allocated_storage(optional)auto_minor_version_upgrade(optional)backup_retention_period(optional)ca_certificate_identifier(optional)copy_tags_to_snapshot(optional)db_instance_arn(optional)db_instance_class(optional)db_instance_port(optional)db_instance_status(optional)db_name(optional)dbi_resource_id(optional)engine(optional)engine_version(optional)enhanced_monitoring_resource_arn(optional)license_model(optional)master_username(optional)monitoring_interval(optional)monitoring_role_arn(optional)multi_az(optional)preferred_backup_window(optional)preferred_maintenance_window(optional)publicly_accessible(optional)secondary_availability_zone(optional)storage_encrypted(optional)storage_type(optional)Field FilteringFields in the response body can be filtered using thefieldsquerystring argument. Fields are acomma-separatedlist of any of the attributes already returned. Tags cannot be filtered on a per-tag basis, though you may choose to include or exclude thetagsattribute entirely.curlhttp://localhost:5000/instances?fields=id,image_nameComplex FiltersFiltering is performedafterretrieving results from the EC2 API. The following are valid filters:exact: performs an exact match on the value of the tag namein-list: splits the tag value by comma and verifies that the passed value is in the resulting listis-null: checks if the value isnullis-true: checks if the value istrueis-false: checks if the value isfalsesubstring: performs a substring match on the value of the tag nameends-with: performs a match usingendswithon the value of the tag namestarts-with: performs a match usingstartswithon the value of the tag namenot-in-list: splits the tag value by comma and verifies that the passed value isnotin the resulting listnot-substring: performs an inverse substring match on the value of the tag namenot-ends-with: performs an inverse match usingendswithon the value of the tag namenot-starts-with: performs an inverse match usingstartswithon the value of the tag nameA simple example is as followscurlhttp://localhost:5000/instances?substring.name=wwwTag FilteringYou can also filter by tags by using thetags.FILTER.TAG_NAMEquerystring pattern as follows:curlhttp://localhost:5000/instances?tags.exact.bootstrapped=true&tags.substring.Name=adminYou may also specify an exact match filter when omitting theFILTERsection like so:curlhttp://localhost:5000/instances?tags.bootstrapped=trueFiltering by the status fieldValidstatusvalues are as follows:pendingrunningshutting-downterminatedstoppingstoppedcurlhttp://localhost:5000/instances?status=runningErrorsIf an invalid querystring argument is passed, ajsonresponse similar to the following will be sent from the service:{detail:"Invalid region querystring argument passed.",status:400,title:"Invalid argument passed"}The following errors are possible:Invalid region querystring argumentInvalid status querystring argumentGeneral EC2ResponseErrorHow it worksThis services usesboto3underneath to provide an api listing all nodes available in a given EC2 account. It is intended to be used as a replacement for hitting the EC2 api directly, as that can be slow at times.CaveatsWhen using the/instancesor/instances/groupendpoints with a format ofdict, the response is keyed by theNametag of the ec2 instance. If you have multiple servers with the same value for theNametag, this may result in “hidden” servers. If a server is not running, it will be hidden by default, otherwise it may overwrite the previous entry. This will both be logged in thehaldanelogging outputandwill be surfaced asmeta.hidden_nodes, regardless of the endpoint being used.