Package: cs.aws-account
cs.aws_accountBoto3 provides nice bindings into the AWS API.  However, when dealing with
a complex environment with many regions and accounts, the default calls can
become cumbersome.  This package provides some convenience layers to
application builders that wish to remove some of the complexities for
common tasks such as account coordination, call limiting, and threaded
access.Why might you want to use this package?You're worried about call rates to the AWS APIYou're developing a multi-threaded application and you'd like to provide
thread-safe global access to boto3.session.Session objects.You'd like to allow runtime configuration to determine which AWS
accounts and regions are acted on by your application.Narrative Application UsageAlthough individual package components are easy enough to understand, as a
whole, things get complex.  To help your introduction, this section provides
quick narratives to component usage.ConfigurationIt all starts with configuration.  In these examples, we'll leverage Yaml and
Jinga (to allow for anchors/references and config macros) to help
iilustrate how a real-world deployment might work.Given the following YAML file, namedconfig.yaml:SessionParameters:&aws_api_credsaws_access_key_id:{{AWS_ACCESS_KEY_ID}}aws_secret_access_key:{{AWS_SECRET_ACCESS_KEY}}Account:&default_accountSessionParameters:*aws_api_credsAssumeRole:RoleArn:{{AWS_ASSUME_ROLE}}RoleSessionName:test_assumed_role_session#arbitrary nameRateLimit:&default_ratelimitmax_count:10interval:1block:TrueRegionalAccounts:&us-east-1Account:*default_accountRateLimit:*default_ratelimitFilter:Partitions:aws:IncludeNonRegional:TrueRegions:include:[us-east-1]RegionalAccounts:&all-execpt-us-east-1Account:*default_accountRateLimit:*default_ratelimitFilter:Partitions:aws:Regions:exclude:[us-east-1]YourApp:RegionalAccountSet:-RegionalAccounts:*us-east-1-RegionalAccounts:*all-execpt-us-east-1The following code will parse the YAML into Python data structures and
interpolate the named environment variables into the configuration.>>>fromjinja2importTemplate>>>importos>>>importyaml>>>withopen('config.yaml')asconfig_file:...rendered_config=Template(config_file.read()).render(os.environ)>>>config=yaml.load(rendered_config)Most of the entries in the config have parameters defined bycs.aws_account.
The exception is theYourAppentry, which determines how your application
will consume the rest of the configuration parameters.  In this example,
the application is choosing to produce acs.aws_account.regional_account_set.RegionalAccountSetinstance for consumption.Here's how the app should initialize this object>>>fromcs.aws_account.regional_account_setimportregional_account_set_factory>>>accounts=None>>>defmy_app_initialization(config):...globalaccounts...accounts=regional_account_set_factory(*config['YourApp']['RegionalAccountSet'])>>>my_app_initialization(config)There is now a globalaccountsvariable that can be leveraged across your
application whose contents was determined via configuration.  Some important
notes about this object:It is iterable, producing instances ofcs.aws_account.regional_account.RegionalAccountobjects.cs.aws_account.regional_account.RegionalAccountobjects have methods to
interface with boto3Session.Client()andSession.Client().get_paginator()that have built-in ratelimiting...this is one of the reasons you're
leveraging this package.the usage ofcs.aws_account.regional_account_set.regional_account_set_factoryinsured that the objects referenced are cached object singletons.  This
means that other config-driven factory based objects with common call
signatures will also reference these singletons.Custom endpointsWith AWS and boto3, customizing the endpoints used for the various AWS services
is possible, though doing so can be a bit finicky. To simplify this process withcs.aws_account, add a nested object to theSessionParametersconfiguration block in your YAML config, like the example below:ServiceEndpoints:&vpc_service_endpointsec2:{{EC2_VPC_ENDPOINT}}sqs:us-east-1:{{SQS_VPC_ENDPOINT_US_EAST_1}}us-west-2:{{SQS_VPC_ENDPOINT_US_WEST_2}}SessionParameters:&aws_api_credsaws_access_key_id:{{AWS_ACCESS_KEY_ID}}aws_secret_access_key:{{AWS_SECRET_ACCESS_KEY}}ServiceEndpoints:*vpc_service_endpointsAccount:&default_accountSessionParameters:*aws_api_credsAssumeRole:RoleArn:{{AWS_ASSUME_ROLE}}RoleSessionName:test_assumed_role_session#arbitrary nameRateLimit:&default_ratelimitmax_count:10interval:1block:TrueRegionalAccounts:&us-east-1Account:*default_accountRateLimit:*default_ratelimitFilter:Partitions:aws:IncludeNonRegional:TrueRegions:include:[us-east-1]YourApp:RegionalAccountSet:-RegionalAccounts:*us-east-1There are two important points to note here:Cross-region custom endpoints are supported by specifying the endpoint as a map of region
aliases to endpoints instead of strings for each service.When using custom endpoints, because of limitations in the underlying boto3 library, the
region specified for a given AWS API call must match the region of the custom endpoint, orcs.aws_accountwill fallback to the default endpoint for that service.ComponentsThe SessionThecs.aws_account.Sessionclass provides convienent, threadlocal access toboto3.session.Sessionobjects.  Major features includes:a singlecs.aws_account.Sessionobject can be used across threads safelymemoized methods for some methods for performant re-calls (i.e. for
logging)built-in support for IAM role-assumption, including automated credential
refreshingCreating acs.aws_account.Sessionis easy...just pass in the same kwargs you
would forboto3.session.Session(in this example, we get the values from the
current environment).>>>importos>>>fromcs.aws_accountimportSession>>>session_kwargs={...'aws_access_key_id':os.environ.get('AWS_ACCESS_KEY_ID'),...'aws_secret_access_key':os.environ.get('AWS_SECRET_ACCESS_KEY')...}>>>session=Session(**session_kwargs)You can now get access to a threadlocalboto3.session.Sessionobject.>>>b3=session.boto3()>>>type(b3)<class'boto3.session.Session'>>>>b3.get_credentials().access_key==session_kwargs['aws_access_key_id']TrueMultiple calls will return the same object (within the same thread)>>>session.boto3()issession.boto3()TrueDifferent threads will get unique threadlocalboto3.session.Sessionobjects>>>importthreading>>>thread_b3=[]>>>defb3():...thread_b3.append(session.boto3())>>>t1=threading.Thread(target=b3)>>>t1.start();t1.join()>>>thread_b3[0]issession.boto3()FalseIt's easy to assume new IAM roles, simply callcs.aws_account.Session.assume_role()with the same arguments asboto3.session.Session().client('sts').assume_role()>>>b4_key=session.access_key()>>>assume_role_kwargs={...'RoleArn':os.environ.get('AWS_ASSUME_ROLE'),...'RoleSessionName':'testing_assume_role_for_cs_aws_account_package'...}>>>session.assume_role(**assume_role_kwargs)>>>b4_key==session.access_key()FalseAssuming a role is threadsafe and will cascade to other threads.  Keep in mind
that other threads will need to get new references to realize this (e.g.
callcs.aws_account.Session.boto3()again)>>>thread_key=[]>>>defb3():...thread_key.append(session.access_key())...new_b3=session.boto3()# just illustrating that threads will need to get a new reference>>>t1=threading.Thread(target=b3)>>>t1.start();t1.join()>>>session.access_key()==thread_key[0]TrueWe can also revert from an assumed role (same as above from a thread standpoint)>>>assumed_role_key=session.access_key()>>>assumed_boto3_session=session.revert()#returns the pop'd session>>>session.access_key()==assumed_role_keyFalse>>>session.access_key()==b4_keyTrueFor environments that desire to leverage singletons for commoncs.aws_account.Session()initialization parameters, there is a convienence
factory>>>fromcs.aws_accountimportsession_factory>>>session_factory(SessionParameters=session_kwargs)is\...session_factory(SessionParameters=session_kwargs)TrueThe caching singleton factories are only semi intelligent when it comes to
understanding common non-hashable argument values.  The internal algorithm
simply iterates args, sorts kwargs, and aggregates thestr()values to
determine the results hash...e.g. hash is impacted by things like value
ordering.The AccountThecs.aws_account.Accountclass is mostly a small wrapper on top ofcs.aws_account.Sessionthat provides accessors to cachable account information.>>>fromcs.aws_accountimportAccount>>>account=Account(session=session)>>>assert(account.account_id())>>>account.session()issessionTrue>>>account.alias()inaccount.aliases()True>>>account.account_id()==account.session().account_id()TrueAs withcs.aws_account.Session, there is a caching singleton factory
available for common initialization parameters.  unlike Account, parameters
are the same as those forcs.aws_account.Session.>>>fromcs.aws_accountimportaccount_factory>>>account_factory(SessionParameters=session_kwargs)is\...account_factory(SessionParameters=session_kwargs)TrueThe Regional AccountThecs.aws_account.RegionalAccountclass provides a thread-safe, runtime
adjustable, region-specific, rate limited boto3 Client caller and
paginator.Seecs.ratelimitfor detailed information on how rate limiting operates
and the various configuration choices.>>>fromcs.aws_accountimportRegionalAccount>>>fromcs.ratelimitimportRateLimitProperties,RateLimitExceeded>>>fromdatetimeimporttimedelta>>>rl=RateLimitProperties(max_count=1,interval=timedelta(seconds=1),block=False)>>>raccount=RegionalAccount(rl,account,region_name='us-east-1')>>>raccount.region()=='us-east-1'True>>>raccount.account()isaccountTrueSimple boto3 Client call rate limits are now available (these calls are always
routed to named region).>>>_=raccount.call_client('sts','get_caller_identity')>>>try:..._=raccount.call_client('sts','get_caller_identity')...exceptRateLimitExceeded:...print('Too fast!')Toofast!We can change the rate limit behavior at runtime for the instance>>>raccount.ratelimit.max_count=2>>>_=raccount.call_client('sts','get_caller_identity')We can also get access to a rate limited paginator.  All calls to boto3 are
contolled by the same instance rate limiter.>>>paginator=raccount.get_paginator('ec2','describe_instances')>>>page_iterator=paginator.paginate(PaginationConfig={'MaxItems':10})>>>try:..._=page_iterator.__iter__().next()...exceptRateLimitExceeded:...print('Too fast!')Toofast!>>>raccount.ratelimit.max_count=3>>>_=page_iterator.__iter__().next()All of this functionality is thread safe>>>raccount.ratelimit.max_count=1>>>raccount.ratelimit.interval=timedelta(seconds=.1)>>>defparallel_call(raccount):...raccount.ratelimit.block=True..._=raccount.call_client('sts','get_caller_identity')>>>t1=threading.Thread(target=parallel_call,args=(raccount,))>>>t1.start()>>>_=raccount.call_client('sts','get_caller_identity')>>>t1.join()As withcs.aws_account.Sessionandcs.aws_account.Account, there is a caching
singleton factory available for common initialization parameters.>>>fromcs.aws_accountimportregional_account_factory>>>kwargs_raccount={...'RateLimit':{'max_count':1,'interval':1,'block':False},...'Account':{'SessionParameters':session_kwargs},...'region_name':'us-east-1'...}>>>regional_account_factory(**kwargs_raccount)is\...regional_account_factory(**kwargs_raccount)TrueRegional Account ContainersTypically, an application has a set of operations that need to be performed
across several AWS accounts and/or regions.cs.aws_account.RegionalAccountsprovides a readonly dict-like interfaces for filtered groups ofcs.aws_account.RegionalAccountobjects.  A couple things to note:TheRegionalAccountsis a read only, thread-safe, container whose contents is
initialized based a filter specification (see below).ARegionalAccountsobject only containsRegionalAccountobjects from a
singleAccount(i.e. from a single AWS account)You can specify a default rate limit that gets individually applied across
contained 'RegionAccount` objects in addition to named-region rate limits.The Container Filter SpecificationThe key to understandingRegionalAccountsis to understand thedictspecification that determines their contents.  The full filter spec is listed
below (inYamlformat).  All keys are optional.Partitions:aws:# valid AWS partition name.  If absent, defaults 'aws'IncludeNonRegional:True|False# include non-regional endpoint names, defaults to FalseRegions:#if absent, defaults to all available regionsinclude:[list,of,regions]#if absent, defaults to all available regionsexclude:[list,of,regions]#takes precedence over includeHere is a sample Python filter (implements above spec):>>>filter1={'Partitions':...{'aws':...{'Regions':...{'include':['us-east-1']}...}}}RegionalAccountscontainers act like read-only dicts>>>fromcs.aws_accountimportRegionalAccounts>>>rl_kwargs={'max_count':1,'interval':1,'block':False}>>>filter1_raccts=RegionalAccounts(rl_kwargs,{'SessionParameters':session_kwargs},Filter=filter1)>>>print([rforrinfilter1_raccts])['us-east-1']>>>'us-east-1'infilter1_racctsTrue>>>isinstance(filter1_raccts['us-east-1'],RegionalAccount)True>>>filter1_raccts['us-east-1'].region()=='us-east-1'True>>>len(filter1_raccts)1>>>print(filter1_raccts.get('us-east-2','not there'))notthere>>>print([kforkinfilter1_raccts.keys()])['us-east-1']>>>isinstance(filter1_raccts.values()[0],RegionalAccount)True>>>len(filter1_raccts.items())==1TrueWe can now compare filtered vs non-filtered containers>>>all_raccts=RegionalAccounts(rl_kwargs,{'SessionParameters':session_kwargs})>>>len(all_raccts)>len(filter1_raccts)True>>>'us-east-2'notinfilter1_racctsTrue>>>len(filter1_raccts)1>>>'us-east-2'inall_racctsTrueFilters can be mutated, but not replaced.  Mutations arenotgenerally
thread-safe, but the most typcail operations are atomic (e.g. thread safe)
seehttp://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm>>>filter1_raccts.filter['Partitions']['aws']['Regions']['include'].append('us-east-2')>>>filter1_raccts.filter['Partitions']['aws']['Regions']['include']=('us-east-1','us-east-2',)>>>'us-east-2'infilter1_racctsTrue>>>try:...filter1_raccts.filter={}...exceptValueError:...print("not allowed!")notallowed!because theRegionalAccountsimplementation leverages the cachingregional_account_factoryfactory to produceRegionalAccountobjects, the
referenced objects in the 2 containers above are the same>>>all_raccts['us-east-1']isfilter1_raccts['us-east-1']TrueThe fact that these containers leverage caching factories to populate their
contents enables common rate-limits to be applied across containers....that's
a feature.As with the previous types, there is a caching singleton factory available for
common initialization parameters.>>>fromcs.aws_accountimportregional_accounts_factory>>>regional_accounts_factory(RateLimit=rl_kwargs,Account={'SessionParameters':session_kwargs})is\...regional_accounts_factory(RateLimit=rl_kwargs,Account={'SessionParameters':session_kwargs})TrueRegional Account SetSo far we've seen aSession,Account,RegionalAccount, andRegionalAccounts.
All instances of these types can act against a single AWS account.  Real
world usage scenarios extend beyond single AWS accounts.  To deal with this,
we provide a new (and final) container type...thecs.aws_account.RegionalAccountSet.  The purpose of this new container is to
provide easy threadsafe, iterable access to aggregated sets ofRegionalAccountsvalues (e.g.RegionalAccountobjects)Creating a group is straight forward.  Just keep in mind thatadd(),discard()andvalues()calls refer toRegionAccountscontainers, whilst__iter__()returnsRegionAccountobjects>>>fromcs.aws_accountimportRegionalAccountSet>>>set1=RegionalAccountSet(all_raccts)>>>len(set1.values())1>>>isinstance(list(set1.values())[0],RegionalAccounts)True>>>len([raforrainset1])>1True>>>isinstance(set1.__iter__().next(),RegionalAccount)TrueNewRegionalAccountscan be added to the group, but the result set of
iteredRegionalAccountobjects are checked for uniqueness>>>set1.add(filter1_raccts)>>>len(list(set1))==len(all_raccts.values())TrueWe can also removeRegionalAccountsfrom the group>>>set1.discard(all_raccts)>>>len(list(set1))==len(filter1_raccts.values())TrueYou can check the group to see whichRegionalAccountscontainers are available>>>[filter1_raccts]==list(set1.values())TrueThere is also a factory available atcs.aws_account.regional_account_set.regional_account_set_factorywhich accepts
an arbritrary number of dicts providing the call signature ofcs.aws_account.regional_account_set.regional_accounts_factory.WE STOP BREACHES