Package: cgcloud-core
PrerequisitesTo install and use CGCloud, you needPython ≧ 2.7.xpipandvirtualenvGitMac OS X:Xcodeand theXcode Command Line Tools(needed during the
installation of cgcloud-core for compiling the PyCrypto dependency)InstallationRead the entire section before pasting any commands and ensure that all
prerequisites are installed. It is recommended to install CGCloud into a
virtualenv. Create a virtualenv and usepipto install
thecgcloud-corepackage:virtualenv ~/cgcloud
source ~/cgcloud/bin/activate
pip install cgcloud-coreIf you getDistributionNotFound: No distributions matching the version forcgcloud-core, try runningpip install--precgcloud-core.If you get an error aboutyaml.hbeing missing you may need to install
libyaml (via HomeBrew on OS X) or libyaml-dev (via apt-get or yum on Linux).If you getAttributeError: 'tuple' object has no attribute 'is_prerelease'you may need to upgrade setuptoolssudo pip install --upgrade setuptoolsIf you getImportError: cannot import name cgcloud_versionyou may need to upgrade virtualenvsudo pip install --upgrade virtualenvIf, on Mountain Lion, you get:clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
error: command 'clang' failed with exit status 1try the following work-around:export CFLAGS=-Qunused-arguments
export CPPFLAGS=-Qunused-argumentsThe installer places thecgcloudexecutable into thebindirectory of
the virtualenv. Before you can invokecgcloud, you have to activate the
virtualenv as shown above. Alternatively, create a per-user bin directory and
symlink thecgcloudexecutable into it:deactivate
mkdir -p ~/bin
ln -snf ~/cgcloud/bin/cgcloud ~/binAfter addingexportPATH="$HOME/bin:$PATH"to your to your~/.profile,~/.bash_profileor~/.bashrc, you won’t need to explicitly activate the
virtualenv before running cgcloud.You should be able to invokecgcloudnow:cgcloud --helpAuto-completion for BashInstall the awesomeargcompletemodule:pip install argcompleteThen add the following command to your~/.profile:eval "$(/absolute/path/to/virtualenv/bin/register-python-argcomplete cgcloud)"ConfigurationAccess keysAsk your AWS admin to setup an IAM account in AWS for you. Log into Amazon’s
IAM console and generate anaccess keyfor yourself. While your IAM username
and password are used to authenticate yourself for interactive use via the AWS
console, the access key is used for programmatic access viacgcloud.Once you have an access key, create~/.botoon you local computer with the
following contents:[Credentials]
aws_access_key_id = PASTE_YOUR_ACCESS_KEY_ID_HERE
aws_secret_access_key = PASTE_YOUR_SECRET_ACCESS_KEY_HEREThe~/.botofile is being deprecated. Consider using~/.aws/credentialsinstead. It is supported by various AWS SDKs and allows for easily switching
between different AWS accounts (profiles):[foo]
aws_access_key_id=PASTE_YOUR_FOO_ACCESS_KEY_ID_HERE
aws_secret_access_key=PASTE_YOUR_FOO_SECRET_KEY_ID_HERE
region=us-west-2

[bar]
aws_access_key_id=PASTE_YOUR_BAR_ACCESS_KEY_ID_HERE
aws_secret_access_key=PASTE_YOUR_BAR_SECRET_KEY_ID_HERE
region=us-west-2To choose an active profile, set theAWS_PROFILEenvironment variable:export AWS_PROFILE=fooEC2 region and availability zoneEdit your~/.profileor~/.bash_profileand add the following line:export CGCLOUD_ZONE=us-west-2aThis configures both the regionus-west-2and the availability zone within
that region:a. Instead ofus-west-2ayou could useus-east-1aor
any other zone in any other EC2 region.Public SSH keyIf you don’t have an SSH key, you can create one using thessh-keygencommand. Do not use the EC2 console to generate a key. This would be insecure
and produce a key that is incompatible with CGCloud.Register your SSH key in EC2 by running:cgcloud register-key ~/.ssh/id_rsa.pubThe above command imports the given public key to EC2 as a key pair (I know,
the terminology is confusing) but also uploads it to S3, see next paragraph for
an explanation. The name of the key pair in EC2 will be set to your IAM user
account name. In S3 the public key will be stored under its fingerprint.If cgcloud complains that thePrivate key file is encrypted, your private
key is probably encrypted with a passphrase (as it should). You need to add the
key to the SSH agent viassh-addwhich should prompt you for the
passphrase. On Mac OS X this can be made more convenient by runningssh-add-Korssh-add-K/path/to/private/keyonce. This will automatically add
the key to the agent every time you log in. The passphrase will be stored in OS
X’s key chain so won’t have to enter it again.Note: Importing your key pair using the EC2 console is not equivalent tocgcloudregister-key. In order to be able to manage key pairs within a
team, CGCloud needs to know the contents of the public key for every team
member’s key pair. But EC2 only exposes a fingerprint via its REST API, not the
actual public key. For this purpose, CGCloud maintains those public keys in a
special S3 bucket. Usingcgcloudregister-keymakes sure that the public
key is imported to EC2anduploaded to that special S3 bucket. Also note that
while that S3 bucket is globally visible and the public keys stored therein
apply across regions, the corresponding key pair in EC2 is only visible within
a zone. So when you switch to a different region, you will have to usecgcloudregister-keyagain to import the key pair into that EC2 region.Multi-user SSH loginsBy default, CGCloud only injects your public key into the boxes that it
creates. This means that only you can SSH into those boxes. If you want other
people to be able to SSH into boxes created by you, you can specify a list of
key pairs to be injected into boxes. You can do so as using the-kcommand
line option tocgcloud createor by setting theCGCLOUD_KEYPAIRSenvironment variable. The latter will inject those key pairs by default into
every box that you create. The default for-kis the special string__me__which is substituted with the name of the current IAM user, i.e.
you. This only works your IAM user account and your SSH key pair in EC2 have
the same name, a practice that is highly recommended. Thecgcloudregister-keycommand follows that convention by default.The most useful shortcut for-kandCGCLOUD_KEYPAIRShowever is to list
the name of an IAM group by prefixing the group name with@@. Assuming that
there exists an IAM group calleddevelopers, adding the following line to
your.profileor.bash_profile:export CGCLOUD_KEYPAIRS="__me__ @@developers"will inject your own key pair and the key pair of every user in thedevelopersIAM group into every box that you create from that point
onwards. Obviously, this only works if EC2 key pairs and IAM usernames are
identical but as mentioned above, if you usedcgcloudregister-keythis
should be the case.In the above example, if a user is removed from the IAM groupdevelopersor
if their key pair is deleted from EC2, his or her key pair will be
automatically removed from every box that was created with that value ofCGCLOUD_KEYPAIRS.Note that a change toCGCLOUD_KEYPAIRSdoes not affect boxes created withcgcloud recreate ROLE. You will need to create a new image usingcgcloud create-ITROLEfor the change to take effect.First stepsYou’re now ready to create your firstboxaka EC2 instance or VM:cgcloud create generic-ubuntu-trusty-boxThis creates a Ubuntu Trusty instance from a stock Ubuntu AMI and then further
customizes it by running additional commands via SSH. It’ll take a few minutes.
Thegeneric-ubuntu-trusty-boxargument denotes arole, i.e. a blueprint
for an instance. You can usecgcloudlist-rolesto see the available roles.Now login to the newly created box:cgcloud ssh generic-ubuntu-trusty-boxThe astute reader will notice that it is not necessary to remember the public
hostname assigned to the box. As long as there is only one box per role, you
can refer to the box by using the role’s name. Otherwise you will need to
disambiguate by specifying an ordinal using the-ooption. Usecgcloud listto view all running instances and their ordinals.Also note that it isn’t necessary to specify the account name of the
administrative user to log in as, e.g.ec2-user,rootorubuntu.
The stock images for the various Linux distributions use different account
names but CGCloud conveniently hides these differences.In order to copy files to and from the box you can usecgcloud rsync:cgcloud rsync generic-ubuntu-trusty-box -av ~/mystuff :Thecgcloud rsynccommand behaves like a prefix to thersynccommand
with one important difference: With rsync you would specify the remote hostname
followed by a colon, withcgcloud rsyncyou simply leave the hostname blank
and only specify a colon followed by the remote path. If you omit the remote
path, the home directory of the administrative user will be used.You can now stop the box withcgcloud stop, start it again usingcgcloud startor terminate it usingcgcloud terminate. Note while a stopped
instance is much cheaper than a running instance, it is not free. Only theterminatecommand will reduce the operating cost incurred by the instance
to zero.If you want to preserve the modifications you made to the box such that you can
spawn another box in the future just like it, stop the box and then create an
image of it using thecgcloud imagecommand. You may then use thecgcloud recreatecommand to bring up a box.Philosophical remarksWhile creating an image is a viable mechanism to preserve manual modifications
to a box, it is not the best possible way. The problem with it is that you will
be stuck with the base image release the box was created from. You will also be
stuck with the customizations performed by the particular version ofcgcloudyou were using. If either the base image or the role definition in
CGCloud is updated, you will not benefit from those updates. Therefore, the
preferred way of customizing a box is byscriptingthe customizations. This
is typically done by creating a CGCloud plugin, i.e. a Python package with VM
definitions akaroles. A role is a subclass of the Box class while a box
(aka VM aka EC2 instance) is an instance of that class. The prominent design
patterns formed by Box and its derived classes areTemplate MethodandMix-in. The mix-in pattern introduces a sensitivity to Python’s method
resolution order so you need to be aware of that.Creating an image makes sense even if you didn’t make any modifications aftercgcloud create. It captures all role-specific customizations made bycgcloud create, thereby protecting them from changes in the role
definition, the underlying base image and package updates in the Linux
distribution used by the box. This is key to CGCloud’s philosophy: It gives you
a way tocreatean up-to-date image with all the latest software according to
your requirementsandit allows you reliably reproduce the exact result of
that step. The fact thatrecreateis much faster thancreateis icing
on the cake.Building & TestingFirst, clone this repository andcdinto it. To run the tests usepython setup.py nosetests--with-doctest,python setup.py test,nosetestorpython-munittest discover-ssrc.We prefer the way listed first as it installs all requirementsandruns the
tests under Nose, a test runner superior tounittestthat can run tests in
parallel and produces Xunit-like test reports. For example, on continuous
integration we usevirtualenv env
env/bin/python setup.py nosetests --processes=16 --process-timeout=900To make aneditableinstall, also known asdevelopment mode, usepython setup.py develop. To remove the editable installpython setup.py develop-u.TroubleshootingIfcgcloud creategets stuck repeatedly printingPrivate key file is encrypted, your private key is probably encrypted with a passphrase (as it
should). You need to add the key to the SSH agent viassh-addwhich
should prompt you for the passphrase. On Mac OS X this can be made more
convenient by runningssh-add-Korssh-add-K/path/to/private/keyonce. This will automatically add the key to the agent every time you log in.
The passphrase will be stored in OS X’s key chain so won’t have to enter it
again.If you get the following error:ERROR: Exception: Incompatible ssh peer (no acceptable kex algorithm)
ERROR: Traceback (most recent call last):
ERROR:   File "/usr/local/lib/python2.7/site-packages/paramiko/transport.py", line 1585, in run
ERROR:     self._handler_table[ptype](self, m)
ERROR:   File "/usr/local/lib/python2.7/site-packages/paramiko/transport.py", line 1664, in _negotiate_keys
ERROR:     self._parse_kex_init(m)
ERROR:   File "/usr/local/lib/python2.7/site-packages/paramiko/transport.py", line 1779, in _parse_kex_init
ERROR:     raise SSHException('Incompatible ssh peer (no acceptable kex algorithm)')
ERROR: SSHException: Incompatible ssh peer (no acceptable kex algorithm)try upgrading paramiko:pip install --upgrade paramikoSee alsohttps://github.com/fabric/fabric/issues/1212CustomizationCGCloud can be customized via plugins. A plugin is a Python module or package
containing two functions:def roles():
   """
   Return a list of roles, each role being a concrete subclass of
   cgcloud.core.box.Box
   """
   return [ FooBox ]

def command_classes():
   """
   Return a list of command classes, each class being a concrete subclass of
   cgcloud.lib.util.Command.
   """
   return [ FooCommand ]If the plugin is a Python package, these two functions need to be defined in
its__init__.py. The box and command classes returned by these two
functions can be defined in submodules of that package.In order to be loaded by CGCloud, a plugin needs to be loadable fromsys.pathand its module path (foo.bar.blah) needs to be mentioned in theCGCLOUD_PLUGINSenvironment variable which should contains a
colon-separated list of plugin module paths.You can also run CGCloud with the--scriptoption and a path to a Python
script. The script will be handled like a plugin, except that it should not
define acommand_classes()function since that function will not be invoked
for a script plugin. In other words, a script plugin should only define roles,
not commands.