Package: scons-remote
scons-remoteThe goal ofscons-remoteis to extend the SCons build tool to support building targets
in remote compute environments (via AWS EC2). Becausescons-remoteis built entirely on
base SCons classes, it achieves this goal while maintaining the robustness/flexibility of
standard SCons builds.Installationpip install scons-remoteUsageThe primary contribution ofscons-remoteis theEnvironmentRemoteclass
which is built on top of the base SConsEnvironmentclass. This class
implements a remote extension of the baseCommandmethod as well as
methods/attributes to manage the remote execution environment and how targets
are built within it.Initializing Remote ConnectionIn order to correctly instantiate a remote compute node via AWS EC2
we need to initialize the connection parameters at the beginning of
theSConstructscript.Theconnection_initializemethod allows us to do this via the
following arguments:client_args: A dictionary of arguments that is passed verbatim toboto3.clientto create an EC2 client.instance_args: A dictionary of arguments that is passed verbatim to
the client'srun_instancesmethod.ssh_args: A dictionary of arguments that is passed verbatim tofabric.connection.Connection.The following shows how a simple remote configuration might be initialized
in theSConstructscript:importosfromscons_remote.environment_remoteimportEnvironmentRemoteenv=EnvironmentRemote(ENV=os.environ)client_args={'region_name':'us-west-2'}instance_args={'ImageId':'ami-xxxxxxxxxx','InstanceType':'t2.large','KeyName':'xxxxxxx','MaxCount':1,'MinCount':1,'InstanceInitiatedShutdownBehavior':'terminate'}ssh_args={'user':'ubuntu','connect_kwargs':{'key_filename':'path/to/private/key.pem'}}env.connection_initialize(client_args,instance_args,ssh_args)Building Targets RemotelyThe base SConsEnvironment.Commandmethod provides a flexible way of building
targets using generic commands. The general formatting of building a target withCommandwould look something like the following:importosenv=Environment(ENV=os.environ)env.Command(target='foo.bar',source='foo_bar.py',action='python $SOURCES $TARGETS')Translating the prior chunk to build the target usingscons-remotewould be quite simple:importosfromscons_remote.environment_remoteimportEnvironmentRemoteenv=EnvironmentRemote(ENV=os.environ)env.CommandRemote(target='foo.bar',source='foo_bar.py',action=env.ActionRemote(cmd='python'))The only substantive difference between these two methods of building targets is thatCommandRemoterequires that the specified action is created using the environment'sActionRemotemethod whereas the baseCommandmethod accepts a string or a callable Python object.Remote ActionsConstructing a remote action is straightforward;ActionRemoteaccepts the following arguments:cmd: A string specifying the command to executecmd_args: A string or list of strings specifying command line arguments
to be passed tocmd.For example, consider the following action passed toCommand:env.Command(...action='python3 -B -v $SOURCES $TARGETS')The same action translated forCommandRemotewould be:env.CommandRemote(...action=env.ActionRemote(cmd='python3',cmd_args=['-B','-v']))Full SConstruct ExampleCombining the initialization step and the target building step described above, the fullSConstructscript for buildingfoo.barwould look as follows:importosfromscons_remote.environment_remoteimportEnvironmentRemoteenv=EnvironmentRemote(ENV=os.environ)client_args={'region_name':'us-west-2'}instance_args={'ImageId':'ami-xxxxxxxxxx','InstanceType':'t2.large','KeyName':'xxxxxxx','MaxCount':1,'MinCount':1,'InstanceInitiatedShutdownBehavior':'terminate'}ssh_args={'user':'ubuntu','connect_kwargs':{'key_filename':'path/to/private/key.pem'}}env.connection_initialize(client_args,instance_args,ssh_args)env.CommandRemote(target='foo.bar',source='foo_bar.py',action=env.ActionRemote(cmd='python'))And voilà, the target has been built via AWS!Force Targets to be Built LocallyIn some cases it may be convenient to create anSConstructfile that invokesCommandRemotebut still have
the ability to force all targets to be built locally. To accomodate this,scons-remoterespects theSCONS_REMOTE_MODEenvironment variable set to'local'. If this variable is unset or any other value,scons-remotewill simply ignore it.To set this in Linux enter the following:exportSCONS_REMOTE_MODE=localAnd in PowerShell:$env:SCONS_REMOTE_MODE="local"⚠️Known Issues/Shortcomings⚠️The following is a short list of known issues/shortcomings when usingscons-remote(more will probably surface):AWS Credentials Timeout: If your AWS credentials that are being passed to boto3 are of the
expiring-after-one-hour variety, long-running pipelines (greater than one hour) will lose the ability to manage
AWS resources and will error out if any targets are attempted to be built after that point. However, any targets
that are already building should continue their build successfully.Forcing Targets to Build Locally: Say you have forced all targets to be built locally (as described above).
When you unset theSCONS_REMOTE_MODEenvironment variable, SCons will not recognize the targets as already
built and will re-build them remotely. This is because, under the hood,scons-remoteis using two different
builder actions which SCons recognizes and instructs that the targets be re-built. The converse action (switching
from remote to local) will have the same behavior.Matching Local and Remote Compute Environments: Currentlyscons-remotedoes no checking of the remote
compute environment to ensure that it has the necessary dependencies/tools required to successfully build the targets.
This falls entirely on the user to ensure that their AMI is compatible with their use-case.ContributorsDaniel MolitorMark HowisonLicensescons-remoteis freely available for non-commercial use under the license provided in LICENSE. To inquire about commercial use, please contactconnect@ripl.org.