Package: dynamof
DynamofA small :fire: interface for more easily making calls to dynamo using boto. No bloated ORM - just functions that make creating the complex objects needed to pass to boto3 quick and easy.Installpip install dynamofBasic FeaturesSimplifyingboto3function APIs (see an example)If you've ever used boto3 directly before you know the pain that can exist trying to write a genericKeyConditionorConditionExpression.dynamofdoes these things for you. It provides simple functions that take common sense arguments and build the complex objects boto3 uses for you.Standardizingboto3error handling (see an example)If you've ever used boto3 directly you know that handling errors is the absolute worst... how much time I've spent googling how to catch this error or that error.... and they're all different!dynamofwraps the calls to boto3, catches all of its errors, inspects them to determine the specific error it represents, and then throws a concrete and documented exception you can catch with a standardtry...except.Its just a librarydynamofis not a framework and its not opinionated.dynamofis simply a collection of deterministic functions that take in arguments and output boto3 command objects. We also provide a small wrapper for executing those boto3 calls behind the scenes if thats not something you want to do yourself. The benefit, is that you can use raw boto3 calls anddynamofcalls right next to each other.dyanmofdoesn't replace boto3, its a simple layer that sits on top to make things easier and more maintainable for you.dynamofwraps theboto3.client('dynamodb')(docs) functions exposing much easier to use api's. It's written in a functional style with the goal to be as useful to anyone in any way as possible. The wrappers around boto3 functions are split into two parts:operationsandrunners. A runner runs a specific operations. The operation contains all the necessary information for a dynamo action to be ran. This means, you don't have to usedynamofto actually interact with dynamo if you don't want to but you could still use it as a utility to more easily generate the complex objects that are passed to boto3 functions.Why Dynamof?If you're using python and dynamo you have 2 options: an ORM like PynamoDB or Boto3. Kudos to the people who made Pynamo, its great, but it really doesn't scale well. And your stuck with the ORM features even if you don't want them. Interacting with Boto3 directly is a pain. With things likeKeyConditions andConditionExpressions being so difficult to easily grasp you end up duplicating a lot of code in your database/repository/DAL layer. This was my experience. In my early day's I used Pynamo. Once I got tired of trying to bend it to my will at scale I started using Boto3 directly. But... this was still annoying. I wanted a non-opinonated library that could sit on top of Botot3 and do the repetitive, annoying to code work for me.dynamofwas born.Whats Supported?See the two lists below for what has been implemented and what hasn't. If your a developer and want to do something thats not done yet its super easy to implement a new operation. Seethe developer guidefor directions.Currently Supported Calls:white_check_mark: Create table [code|docs|boto3]:white_check_mark: Describe table [code|docs|boto3]:white_check_mark: Add item [code|docs|boto3]:white_check_mark: Find item [code|docs|boto3]:white_check_mark: Update item [code|docs|boto3]:white_check_mark: Delete item [code|docs|boto3]:white_check_mark: Query table [code|docs|boto3]:white_check_mark: Scan table [code|docs|boto3]Currently Unsupported Calls:x:batch_get_item[boto3]:x:batch_write_item[boto3]:x:can_paginate[boto3]:x:create_backup[boto3]:x:create_global_table[boto3]:x:delete_backup[boto3]:x:delete_table[boto3]:x:describe_backup[boto3]:x:describe_continuous_backups[boto3]:x:describe_contributor_insights[boto3]:x:describe_endpoints[boto3]:x:describe_global_table[boto3]:x:describe_global_table_settings[boto3]:x:describe_limits[boto3]:x:describe_table_replica_auto_scaling[boto3]:x:describe_time_to_live[boto3]:x:generate_presigned_url[boto3]:x:get_paginator[boto3]:x:get_waiter[boto3]:x:list_backups[boto3]:x:list_contributor_insights[boto3]:x:list_global_tables[boto3]:x:list_tables[boto3]:x:list_tags_of_resource[boto3]:x:restore_table_from_backup[boto3]:x:restore_table_to_point_in_time[boto3]:x:tag_resource[boto3]:x:transact_get_items[boto3]:x:transact_write_items[boto3]:x:untag_resource[boto3]:x:update_continuous_backups[boto3]:x:update_contributor_insights[boto3]:x:update_global_table[boto3]:x:update_global_table_settings[boto3]:x:update_table[boto3]:x:update_table_replica_auto_scaling[boto3]:x:update_time_to_live[boto3]Getting StartedExample: Create a table in dynamofromboto3importclientfromdynamof.executorimportexecutefromdynamof.operationsimportcreateclient=client('dynamodb',endpoint_url='http://localstack:4569')execute(client,create(table_name='users',hash_key='username'))First thing to note...execute(client, some_operation(...))isn'tsexy... and as engineerssexyis important. Becausedynamofis a simple functional utility library its very easy to bend it into any api you would like.Example: Customize the way you call dynamofKeep it functionalfromfunctoolsimportpartialfromboto3importclientfromdynamof.executorimportexecutefromdynamof.operationsimportcreatefromdynamof.attributeimportattrclient=client('dynamodb',endpoint_url='http://localstack:4569')db=partial(execute,client)# Now calling looks likedb(create(table_name='users',hash_key='username'))db(find(table_name='users',key={'username':'sunshie '}))db(update(table_name='users',key={'username':'sunshie'},attributes={'roles':attr.append('admin'),'friends':attr.prepend('jake')}))Make it a classclassDB:def__init__(self):self.client=client('dynamodb',endpoint_url='http://localstack:4569')deffind(*args,**kwargs):returnexecute(self.client,find(*args,**kwargs))db=DB()db.find(table_name='users',key={'id':21})Make it a table specific classclassTable:def__init__(self,table_name):client=client('dynamodb',endpoint_url='http://localstack:4569')self.table_name=table_nameself.db=partial(execute,client)deffind(*args,**kwargs):returnself.db(find(self.table_name,*args,**kwargs))defupdate(*args,**kwargs):returnself.db(update(self.table_name,*args,**kwargs))defdelete(*args,**kwargs):returnself.db(delete(self.table_name,*args,**kwargs))users=Table('users')users.find(key={'id':21})users.update(key={'id':21},attributes={'username':'new_username_1993_bro'})users.delete(key={'id':21})Example: Catch errors from dynamoffromdynamof.exceptionsimport(UnknownDatabaseException,ConditionNotMetException,BadGatewayException,TableDoesNotExistException)try:db(update(table_name='users',key={'id':43},attributes={'username':'sunshie'}))exceptTableDoesNotExistException:# Handle case where table doesn't existexceptConditionNotMetException:# Handle case where the condition wasn't met (the item you tried to update didn't exist)exceptBadGatewayException:# Handle a network errorexceptUnknownDatabaseException:# Handle an unknown issueExample: Use dynamof to build boto3 arguments but still call boto3 yourselffromdynamofimportoperationsfromdynamof.conditionsimportattrquery=operations.query(table_name='books',conditions=attr('title').equals('The Cost of Discipleship'))result=client.query(**query.description)API Documentationdynamof.operationsdynamof.conditionsOperationsSee the codeSee the testCreate Tablecreate(table_name,hash_key,allow_existing=False)See boto3 docsParameterRequiredData TypeDescriptionExampletable_nameyesstrThe name to assign the table your creating'users'hash_keyyesstrThe hash key (primary key) for your table'user_id'allow_existingnoboolCreating a table that already exists will throw an error in boto3. PassingTruehere will ignore that error if its raised and ignore it.True:orange_book: LimitationsCannot specify range keyCannot specify complex hash key (hash key and range key)Cannot specify indexesOther boto3 parameters not implemented (BillingMode,ProvisionedThroughput,StreamSpecification,SSESpecification,Tags)Find Itemfind(table_name,key)See boto3 docsParameterRequiredData TypeDescriptionExampletable_nameyesstrThe name of the table to find an item in'users'keyyesstr|dictThe key (primary key) of the item to find. If a string is passed it is associated withidby default. If an object is passed the first key and value are used to find the item22or{ 'username': 'sunshie' }:orange_book: LimitationsCannot use projection expressionsOther boto3 parameters not implemented (ConsistentRead,ReturnConsumedCapacity)Add Itemadd(table_name,item,auto_inc=False)See boto3 docsParameterRequiredData TypeDescriptionExampletable_nameyesstrThe name of the table to add the item to'users'itemyesdictThe item to be added to the table in key value pairs. Ifauto_incis not set to true then this dictmustinclude a valid key value pair for the table's hash key{ 'username': 'sunshie', 'user_status': 'unleashed' }:orange_book: Limitationsboto3 parameters not implemented:ReturnItemCollectionMetrics,ReturnConsumedCapacity,ReturnValues:closed_book: Known IssuesWhen the table you're trying to add to does not exist theput_itemfunction in boto3 does not throw the expectedClientErrorwith the table not found code and message. Instead, it throws a bad gateway error. So, when callingaddyou cannot depend on theTableDoesNotExistException. In a future version you will be able to use an additonal method to check if the table exists if needed.Update Itemupdate(table_name,key,attributes)See boto3 docsParameterRequiredData TypeDescriptionExampletable_nameyesstrThe name of the table to update the item on'users'keyyesstr|dictThe key (primary key) of the item to find for updating. If a string is passed it is associated withidby default. If an object is passed the first key and value are used to find the item22or{ 'username': 'sunshie' }attributesyesdictThe key values patch/set on the record{ 'rank': 23 }:orange_book: LimitationsCannot allow setting parameters forReturnValues,ReturnConsumedCapacity,ReturnItemCollectionMetricsDelete Itemdelete(table_name,key)See boto3 docsParameterRequiredData TypeDescriptionExampletable_nameyesstrThe name of the table to delete the item from'users'keyyesstr|dictThe key (primary key) of the item to delete. If a string is passed it is associated withidby default. If an object is passed the first key and value are used to find the item22or{ 'username': 'sunshie' }:orange_book: LimitationsCannot allow setting parameters forReturnValues,ReturnConsumedCapacity,ReturnItemCollectionMetricsQuery Tablequery(table_name,conditions)See boto3 docsParameterRequiredData TypeDescriptionExampletable_nameyesstrThe name of the table to execute the query on'users'conditionsyesdynamof.conditions.ConditionThis value should be built using thedynamof.conditionsmodule. See the docs on that module.attr('username').equals('sunshie')will build a proper Condition to pass.:orange_book: LimitationsCannot do paginationCannot set limitsCannot query indexesCannot allow setting parameters forReturnValues,ReturnConsumedCapacity,ReturnItemCollectionMetricsConditionsThedynamof.conditionsmodule provides utility methods that make it simple to generate the complex data object boto3 needs when specifying conditions for querying, scanning, and other operations. Looking at thedocsfor the query function you'll seeKeyConditionExpression. This is the parameter this module was created to build.Examplefromdynamof.conditionsimportattrcond=attr('username').equals('sunshie')cond.expression# 'username = :username'cond.attr_values# { ":username": { "S": "sunshie" } }attr(name)ParameterRequiredData TypeDescriptionExamplenameyesstrThe name of the attribute to begin using on a condition. Could be a column you want to match exactly or if its a number type then it could be a column you want to check for>or<on'username'Theattrfunction returns adynamof.conditions.Attributethat contains three methodsequals(value)greater_than(value)less_than(value)Where value is always the value to use in the conditional comparison you build.cand(*conditions)Takes any number of condition expressions and combines them using theandrule.ParameterRequiredData TypeDescriptionExampleconditionsyes*dynamof.conditions.ConditionTakes any number ofConditioninstancescand(attr('points').less_than(50)Developer GuideAs a developer you can probably guide yourself, you just want to know in simple terms "how it works". Thats what I'll talk about here - the design, what calls what, what does what. After reading the developer guide you should feel comfortable making changes.The DesignThe goal whendynamofwas started was to create the complicated objects that boto3 takes in as arguments to its calls. It wasn't until later that a wrapper for executing those calls insidedynamofwas added so we could standardize wild boto3 exceptions for the client. This means, at the core ofdynamofisoperationsandbuilder- its that simple. The builder is (for the most part) a DTO (with some sugar) for containg all the arguments you specify for a given operation. If you look in any operation file you'll find the same pattern:defoperation_name(table_name,key,conditions=None):# Build is a function we can use to create boto3 arguments with the details# we pass to `ab.builder`. For example, `build(ab.TableName)` will return the# `table_name` we pass the builder here.build=ab.builder(table_name=table_name,key=key,conditions=conditions)# Description is the object that will get passed to the boto3 call. See `run` below.description=shake(TableName=build(ab.TableName),Key=build(ab.Key),ConditionExpression=build(ab.ConditionExpression),ExpressionAttributeValues=build(ab.ExpressionAttributeValues))returnOperation(description,run)defrun(client,description):# This is where boto3 will be called - if so desired. In the executor this function gets wrapped# in error handling. Again, a pure function (ish - because were making a network call - but as# pure as pure can be for a network calling function).res=client.operation_name(**description)returnresponse(res)# Returns a standard response objectYou can see, this doesn't call dynamo or boto3, its a deterministic/pure function that takes in arguments and uses the builder to generate all the arguments for the boto3 call.How it effects testsSince the operation functions return anOperationwe can write concise, full coverage tests by calling the operation function with different arguments and then looking in on thedescriptionit returned in side theOperation. Heres an example:deftest_operation_creates_description_with_table_name():res=operation_name(table_name='users',key={'username':'sunshie '})assertres.description['TableName']=='users'How to add an operationGiven the information above, here is a checklist you might use when addding a new operation. For example sake, lets say the operation name isdeploy.Create a file for the newdeployoperation atdynamof/operations/deploy.pyDefine two functions inside this filedeploy(...)andrun(client, description)Add any arguments you might need to thedeployfunctionUse the builder to generate all the arguments boto3 expects given thedeployfunction arguments.If needed, go to the builder, and add/modify the functions that create description attributes.Implement therunmethod. The executor expects allrunfunctions to take aclientanddescriptionand return aresponse. This should always be a stupid simple function that just calls boto3 and returns the result.
You have a new operation you can use!