Package: dynamo-objects
dynamo_objectsdynamo_objects is a set of tools to work with DynamoDB in python.It is based onbotoand provides following features:a simple object mapper - use object notation to work with dynamo recordsnew tables are automatically created in the database, so you just write
and deploy the new codetransparent support for table prefixes (multiple databases or multiple environments), you don’t need to handle table prefixes in code, just set the prefix during the database connectionsimple in-memory dynamodb mock for fast unit testssupportsDynamoDB localfor slower testsin-memory cached tables to speedup computational operations on top of DynamoDB - all data is read only once and then results are flushed back in a batchadditional tools - copy data from table to table, a context manager to update table throughputs and set back once operation is completedDiscussion groupInstallation$pipinstalldynamo_objectsDB Connection and Table PrefixesDatabase connection method adds table prefix support on top of boto’s connect_to_region method.
Using the table prefix it is possible to switch the application to different set of tables for different environments (or just use different prefixes for different applications).Use the following snippet to connect to the database:fromdynamo_objectsimportDynamoDatabaseDynamoDatabase().connect(region_name='your-aws-region-name-here',aws_access_key_id='access-key-id',aws_secret_access_key='secret-access-key',table_prefix='my_prefix_')Region name, and aws credentials are passed to the boto’s connect_to_region method, so you can use other wayssuppored by bototo specify aws credentials.
For example, it is not necessary to specify access key id and secret if you use IAM role.Thetable_prefixparameter is used to prefix all the table names with the string you specify.Like if you set the table_prefix tostaging_, the application will use tables likestaging_userandstaging_post. And if you set the prefix todev_then application will usedev_user,dev_post.If you leave the table_prefix empty then it will be justuserandpost.
This way you can easily switch your application from one set of tables to another for different environments (development, staging, production).To connect to the DynamoDB Local, specify the region_name=’localhost’:fromdynamo_objectsimportDynamoDatabaseDynamoDatabase().connect(region_name='localhost',table_prefix='dev_')Object MapperTo use the object mapper, define record and table objects for each DynamoDB table:fromboto.dynamodb2.fieldsimportHashKey,RangeKeyfromdynamo_objectsimportDynamoRecord,DynamoTableclassStore(DynamoRecord):def__init__(self,**data):# define table fields in the __init__ methodself.store_id=''self.name=''self.tags=[]super(Store,self).__init__(**data)classStoreTable(DynamoTable):def__init__(self):super(self.__class__,self).__init__('store',schema=[HashKey('store_id')],throughput={'read':3,'write':3},record_class=Store)Here theStoreTableclass describes the table: table name, schema (hash and optionally range keys), throughput and record class.And theStoreclass describes the table row,
in the__init__method we put all the table fields.See more examples of table/record objects in thetests/schema.pyfile.Now the record object can be created and used like this:store=StoreTable()store=Store()store.name='My Store'table.save(store)# or initialize the fields using the constructorstore2=Store(name='My Store 2')# change the namestore2.name='Another Store'StoreTable().save(store)Compare this to the pure boto code where you have a dictionary-like interface:store=Item(stores,data={name='My Store'})# ....store['nmae']='Another Store'If you mistype the field name like instore['nmae']there will be no error - you will just create a new field in the database.
The main purpose of the object mapper is to prevent this.TheDynamoRecordobject will raise an exception if you mistype the field name.
To actually go schema-less, it is possible to override the_freeze_schemamethod withpassin the method body.You can also override the_check_datamethod to do additional transformations before saving to the database (like convert data types or normalize/unify data format).Find a record, update it and save:table=MyTable()# will raise ItemNotFound exception if record does not existrecord=table.get('my_hash','my_range')record.some_field=100table.save(record)# to handle the case when there is no record int the database use# try/exceptfromboto.dynamodb2.exceptionsimportItemNotFoundtry:record=table.get('my_hash','my_range')exceptItemNotFound:# handle the record not found case# ...# sometimes it is more convenient to get None for non-existing record# `find` method will return None if record does not existrecord=table.find('my_hash','my_range')ifrecordisnotNone:record.some_field=100table.save(record)# get a record or create new one if record does not existrecord=table.get('my_hash','my_range',create=True)record.some_field=200table.save(record)# delete the existing record# `delete` method will return the deleted record, so the record data can be# used for some additional actions like loggingrecord=table.delete('hash','range')Thecreate=Trueoption for thetable.get()method is useful when you want to read the data from the database or get the Null object if data is not found.
For example:classUser(DynamoRecord):def__init__(self,*data):self.name='guest'self.password=''super(User,self).__init__(**data)# Find the user in the database, if not found - the `user` object# will represent guest useruser=table.get(user_id,create=True)# print user name or 'guest' (default)printuser.nameQuery and scan methods have the same interface as boto’squery_2andscan, but will convert the resulting record set intoDynamoRecordobjects.# parameters are the same as for boto's query_2# returns array of records# don't use when you expect a lot of data, because it will# fetch all the data from the database and convert to DynamoRecord# before returningrecords=table.query(hash__eq='value',range__gte=50)...records=table.scan(some_field__gte=10)...# get countcount=table.query_count(hash__eq='value',range__gte=50)Table object also supports the atomic counter update:# increment the `counter_name` field by 2 for the# item with hash key = `myhashkey`table.update_counter('myhashkey',counter_name=2)# decrement the `counter_name` field by 2 for the# item with hash key = `myhashkey` and rangekey = 'myrange'table.update_counter('myhashkey','myrange',counter_name=-2)And it is possible to use boto’s objects directly:table=MyTable()# the boto Table objectboto_table=table.table# ...record=table.get('my_hash','my_range')# the boto Item objectboto_item=record._item# ...Memory tablesMemory tables can be used to cache DynamoDB access in-memory.
Every record is only read once and no data is written until you call thesave_dataorsave_data_batchmethod.# StoreTable is a regular table definition, DynamoTable subclassfrommyschemaimportStoreTablefromdynamo_objects.memorydbimportMemoryTableclassStoreMemoryTable(MemoryTable):def__init__(self):super(StoreMemoryTable,self).__init__(StoreTable())Here we define aStoreMemoryTableclass for in-memory table which wraps theStoreTable(a regular table definition).
Now we can do this:table=StoreMemoryTable()# read records with store_id = 1 and store_id = 2record=table.get(1)record2=table.get(2)# data is not actually saved yet, no write db operationstable.save(record)table.save(record2)# ...# read same records again - will fetch from memory, no db read operationsrecord=table.get(1)record2=table.get(2)# ...# data is not actually saved yet, no write db operationstable.save(record)table.save(record2)# Now we flush all the data back to DynamoDB# the `save_data_batch` will use the `batch write` DynamoDB operationtable.save_data_batch()This can be very useful if you do some computational operations and need to read / write a lot of small objects to the database.
Depending on the data structure the used read / write throughput and the whole processing time can be noticeably reduced.Testing and DynamoDB MockIt is possible to run unit tests using the real DynamoDB connection using the table prefixes feature: you can choose some special table prefix likexx_unit_tests_.
This way you’ll have a set of tables for your unit tests.But this approach is not practical - tests will be slow and will consume the read/write operations (and this will cost money).Amazon provides aDynamoDB emulator in javabut it is problematic to use it during development, because it is slow and consumes a lot of memory.The solution is a simple in-memoryDynamoDB mock module.
It is a fast, but very approximate dynamo emulation without permanent data storage.To enable the mock, just import thedynamockmodule:fromdynamo_objectsimportdatabase# once imported, the `dynamock` module will mock real DynamoDB# operations and forward them to the simple implementation which# keeps all the data in memoryfromdynamo_objectsimportdynamockThere is an example of the mock usage in thetests/base.pymodule.This base test module can be used for any project to test parts of code which work with DynamoDB.
You can find examples of unit tests under thetestsfolder. The database schema is described in thetests/schema.py.To run all tests usenosetests(install withpip install nose):nosetestsBy default it will use the in-memoryDynamoDB mock.
To run tests against the  DynamoDB Local use following commands:# in the first terminal window launch the local dynamodb
# script will download it if necessary./tool/dynamodb-local.sh# in another terminal window run the testsDYNAMODB_MOCK=nosetestsI use fast in-memory mock to run tests locally, during the development.On the CI server tests a launched two times - first against the in-memory mock and then one more time against the DynamoDB local.Here is an example of the shell script to do this:# Run fast tests with in-memory mocknosetestsRESULT_MOCK=$?# Run slow tests with DynamoDB localpushdpath/to/folder/with/dynamodb-localjava-Djava.library.path=./DynamoDBLocal_lib-jar./DynamoDBLocal.jar-inMemory-sharedDb&PID=$!popdecho"Started local dynamodb:$PID"DYNAMODB_MOCK=nosetestsRESULT_LOCALDB=$?kill-9$PIDexit$(($RESULT_MOCK+$RESULT_LOCALDB))Additional ToolsThedatabasemodule contains few additional useful tools.Thecopy_itemandcopy_table_datamethods allow to copy data from table to table (for example, you may want to copy some data from staging to production):db=database.Database()# note: table_prefix is empty, so we can explicitly set table namesdatabase.connect(region_name='...',...table_prefix='')num_copied=db.copy_table_data('table_name','staging_table_name')# copy and transform datadeftransform_fn(record):record.name='staging_'+record.namedb.copy_table_data('table_name','staging_table_name',transform=transform_fn)There are also some other useful methods to create the table, wait until the new table becomes active, delete the table, etc.TheTableThroughputclass is a context manager to update (usually set higher) throughput limits and put them back after some operation.
It is useful when you need to do something what requires a high read/write throughput.Using theTableThroughputit is possible to set high limits just before the operation and set them back just after it:high_throughputs={'table_one':{'table':{'read':100,'write':50,},},'table_two':{'table':{'read':60,'write':120,},'SomeGlobalIndex':{'read':1,'write':120}}}withdatabase.TablesThroughput(high_throughputs):# now throughputs are highsome_comutational_operation()# now throughputs are low again (same as before the operation)Related projectsflywheel- Object mapper for Amazon’s DynamoDBPynamoDB- A pythonic interface to Amazon’s DynamoDBDynamodb-mapperDynamodb-mapper - a DynamoDB object mapper, based on boto