Package: marchitect
MarchitectA tool for uploading files to and running commands on remote hosts.Install$pip3installmarchitectExampleLet's write a couple of files to your machine assuming that it could easily be
made to be a remote machine.frommarchitect.site_planimportSitePlan,Stepfrommarchitect.whiteprintimportWhiteprintclassHelloWorldWhiteprint(Whiteprint):name='hello_world'def_execute(self,mode:str)->None:ifmode=='install':# Write file by running remote shell commands.self.exec('echo "hello, world." > /tmp/helloworld1')# Write file by uploadingself.scp_up_from_bytes(b'hello, world.','/tmp/helloworld2')classMyMachine(SitePlan):plan=[Step(HelloWorldWhiteprint)]if__name__=='__main__':# SSH into your own machine, prompting you for your password.importgetpassimportosuser=os.getlogin()password=getpass.getpass('%s@localhost password: '%user)sp=MyMachine.from_password('localhost',22,user,password,{},[])# If you want to auth by private key, use the below:# (Note: The password prompt will be for your private key, empty for none.)#sp = MyMachine.from_private_key(#    'localhost', 22, user, '/home/%s/.ssh/id_rsa' % user, password, {}, [])sp.install()# Sets the mode of _execute() to install.This example requires that you can SSH into your machine via password. To use
your SSH key instead, uncomment the lines above. After execution, you should
have/tmp/helloworld1and/tmp/helloworld2on your machine.Hopefully it's clear that whiteprints let you run commands and upload files to a
target machine. A whiteprint should contain all the operations for a common
purpose. A site plan contains all the whiteprints that should be run on a
single machine class.Steps for deploying your code repository to a machine would make for a good
whiteprint. A site plan for a machine that runs your web servers might use that
whiteprint and others.GoalsEasy to get started.Templating of configuration files.Mix of imperative and declarative styles.Arbitrary execution modes (install, update, clean, start, stop, ...).Interface for validating machine state.Be lightweight because most complex configurations are happening in
containers anyway.Non-goalsMaking whiteprints and site plans share-able with other people and companies.Non-Linux deployment targets.ConceptsWhiteprintTo create a whiteprint, extendWhiteprintand define anameclass variable
and an_execute()method; optionally define avalidate()method.nameshould be a reasonable name for the whiteprint. In the example above, theHelloWorldWhiteprintclass's name is simplyhello_world.nameis
important for file resolution which is discussed below._execute()is where all the magic happens. The method takes a string calledmode. Out of convention, your whiteprints should handle the following modes:install(installing software)update(updating software)clean(removing software, if needed, but generally impractical)start(starting services)stop(stopping services).Despite this convention,modecan be anything as you'll be choosing the modes
to execute your site plans with.Within_execute(), you're given all the freedom to shoot yourself in the
foot. Useself.exec()to run any command on the target machine.exec()returns anExecOutputobject with variablesexit_status(int),stdout(bytes), andstderr(bytes). You can use these outputs to control
flow. If the exit status is non-zero, aRemoteExecErroris raised. To
suppress the exception, seterror_ok=True._execute()has access toself.cfgwhich are the config variables for the
whiteprint. See the Templates & Config Vars section below.Use the variety of functions to copy files to and from the host:scp_up()- Upload a file from the local host to the target.sp_up_from_bytes()- Create a file on the target host from the bytes arg.scp_down()- Download a file from the target to the local host.scp_down_to_bytes()- Download a file from the target and return it.Templates & Config VarsYou can upload files that arejinja2templates. The
templates will be filled by the config variables passed to the whiteprint.
Config variables can be set in a few ways, which we'll explore.Here's a sampletest.tomlfile that uses the jinja2 notation to specify anamevariable with a default ofJohn Doe:name="{{ name|default('John Doe') }}"A whiteprint can populate a template for upload as follows:frommarchitect.whiteprintimportWhiteprintclassWhiteprintExample(Whiteprint):default_cfg={'name':'Alice'}def_execute(self,mode:str)->None:ifmode=='install':self.scp_up_template('/path/to/test.toml','~/test.toml')A whiteprint can also upload a populated template that's stored in a string
rather than a file:frommarchitect.whiteprintimportWhiteprintclassWhiteprintExample(Whiteprint):default_cfg={'name':'Alice'}def_execute(self,mode:str)->None:ifmode=='install':self.scp_up_template_from_str('name = "{{ name }}"','~/test.toml')A config var can be overriden inscp_up_template_from_str:frommarchitect.whiteprintimportWhiteprintclassWhiteprintExample(Whiteprint):default_cfg={'name':'Alice'}def_execute(self,mode:str)->None:ifmode=='install':# 'Bob' overrides 'Alice'self.scp_up_template_from_str('name = "{{ name }}"','~/test.toml',cfg_override={'name':'Bob'})Config vars can also be set by theSitePlanin the plan or during
instantiation.frommarchitect.site_planimportStep,SitePlanclassMyMachine(SitePlan):plan=[Step(WhiteprintExample,{'name':'Eve'})]if__name__=='__main__':MyMachine.from_password(...,cfg={WhiteprintExample:{'name':'Foo'}})In the above,Footakes precedence overEvewhich takes precedence over any
values fornamedefined in the whiteprint.Config Override by AliasFinally, aStepcan be given an alias as another identifier for specifying
config vars. This is useful when a whiteprint is used multiple times in a site
plan.frommarchitect.site_planimportStep,SitePlanclassMyMachine(SitePlan):plan=[Step(WhiteprintExample,alias="ex1"),Step(WhiteprintExample,alias="ex2"),]if__name__=='__main__':MyMachine.from_password(...,cfg={'ex1':'Eve','ex2':'Foo'})In the above, the firstWhiteprintExampleuploadsEveand the second
replaces it withFoo.Auto-Derived ConfigsAuto-derived config variables are always available without specification.
These are stored inself.cfg['_target']:user: The login user for the SSH connection.host: The target host for the SSH connection.kernel: The kernel version of the target host. Ex:4.15.0-43-genericdistro: The Linux distribution of the target host. Ex:ubuntudisto_version: The version of the Linux distribution. Ex:18.04hostname: The hostname of the target host.fqdn: The fully-qualified domain name of the target host.cpu_count: The number of CPUs on the target host. Ex:8Config Var SchemaBecause config vars may be used in external template files, it's not readily
observable what vars are used by a whiteprint. To make config vars explicit,
a schema can be set usingcfg_schema:frommarchitect.whiteprintimportWhiteprintimportschemaclassWhiteprintExample(Whiteprint):cfg_schema={'name':str,schema.Optional('path'):str,'targets':[str],}...The schema is enforced on execution of the whiteprint.For more info on expressing schemas (nesting, lists, optionals), seeschema.File ResolutionMethods that upload local files (scp_up()andscp_up_template()) will
search for the files according to thersrc_pathsargument in theSitePlanconstructor. The search proceeds in order of thersrc_pathsand the name of
the whiteprint is expected to be the name of a subfolder in thersrc_path.For example, assumersrc_pathsis[Path('/srv/rsrcs')], the whiteprint
has a name offoobar, and the filecis referenced asa/b/c. The resolver
will look for the existence of/srv/rsrcs/foobar/a/b/c.If a file path is specified as absolute, say/a/b/c, norsrc_pathwill be
prefixed. However, this form is not encouraged for portability across machines
as resources may live in different folders on different machines.IdempotenceIt's important to strive for the idempotence of your whiteprints. In other
words, assume your whiteprint in any mode (install, update, ...) can be
interrupted at any point. Can your whiteprint be re-applied successfully
without any problems?If so, your whiteprint is idempotent and is therefore resilient to connection
errors and software hiccups. Error handling will be as easy as retrying your
whiteprint a bounded number of times. If not, you'll need to figure out an
error handling strategy. In the extreme case, you can terminate servers that
produce errors and start over with a fresh one, assuming that you're in a cloud
environment.PrefabPrefabs are built-in, robust whiteprints you can use in your whiteprints.
These make it easy to add common functionality with the_execute()and_validate()methods already defined. These are available out-of-the-box:Apt: Common Linux package manager.Pip3: Python package manager.Folder: Makes a folder exists at the specified path.LineInFile: Ensures the specified line exists in the specified file.FileFromString: Makes a file at a specified path.FileFromPath: Makes a file at a specified path.Symlink: Makes a symlink.FileExistsValidator: Only validates that a file exists at a specified path.An example:frommarchitect.prefabimportAptfrommarchitect.whiteprintimportPrefab,WhiteprintclassHelloWorld2Whiteprint(Whiteprint):prefabs_head=[Prefab(Apt,{'packages':['curl']}),]def_execute(self,mode:str)->None:ifmode=='install':self.exec('curl https://www.nytimes.com > /tmp/nytimes')prefabs_headare applied before your_execute()and_validate()methods,
respectively. Alternatively,prefabs_tailare applied after.If a prefab depends on a config variable, define a_compute_prefabs_head()class method:fromtypingimportAny,Dict,Listfrommarchitect.prefabimportFolderfrommarchitect.whiteprintimportPrefab,WhiteprintclassExampleWhiteprint(Whiteprint):cfg_schema={'temp_folder':str,}@classmethoddef_compute_prefabs_head(cls,cfg:Dict[str,Any])->List[Prefab]:return[Prefab(Folder,{'path':cfg['temp_folder']})]The prefabs returned by_compute_prefabs_head()will be applied after those
specified directly in theprefabs_headclass variable.Nested WhiteprintsWhiteprints can use other whiteprints.frommarchitect.whiteprintimportWhiteprintclassExample2Whiteprint(Whiteprint):passclassExampleWhiteprint(Whiteprint):def_execute(self,mode:str)->None:ifmode=='install':self.use_execute(mode,Example2Whiteprint,{})def_validate(self,mode:str)->None:ifmode=='install':self.use_validate(mode,Example2Whiteprint,{})Site PlanSite plans are collections of whiteprints. You likely have distinct roles for
the machines in your infrastructure: web hosts, api hosts, database hosts, ...
Each of these should map to their own site plan which will install the
appropriate whiteprints (postgres for database hosts, uwsgi for web hosts, ...).TestingTests are run against real SSH connections, which unfortunately makes it
difficult to run in a CI environment. Travis CI is not enabled for this reason.
When running tests throughpy.testortox, you can specify SSH credentials
either as a user/pass pair or user/private_key. For example:SSH_USER=username SSH_HOST=localhost SSH_PRIVATE_KEY=~/.ssh/id_rsa SSH_PRIVATE_KEY_PASSWORD=*** py.test
SSH_USER=username SSH_HOST=localhost SSH_PASSWORD=*** py.test -sxWill likely move to mocking SSH commands, but it will be painful to reliably
mock the interfaces forssh2-python.mypyandlintare also supported:tox -e mypy,lintTODOAdd "common" dependencies to minimize invocations of commands likeapt updateto once per site plan.Write a log of applied site plans and whiteprints to the target host
for easy debugging.Add documentation forvalidate()method.Verify speed wins by usingssh2-pythoninstead ofparamiko.DocumentSitePlan.one_off_exec().File prefabs can use md5sum to decide whether to re-create file.