Package: pydantic-dynamo
pydantic-dynamoA Python repository over DynamoDB, leveraging the excellentPydanticlibrary to model records.ContributingPlease seethe contributing guide.InstallationInstall from PyPI:pip install pydantic-dynamoOr even better, usePoetry:poetry add pydantic-dynamoUsageAll of the goodness of pydantic data modeling, pointed at your DynamoDB table instead of your REST API.The ideal consumer is a developer of a low-to-medium complexity application. You will lose
some benefits of the single-table pattern, specifically the ability to query
and retrieve objects of different types in a single connection. Except the most complex single-table examples,
access patterns can be implemented to utilize thelist/list_betweenandget_batchfunctions, documented below, to prevent N+1 queries and provide sufficient performance.Table CreationThis package assumes the specified table already exists and the application it is
running from has sufficient access to the table.The following IAM permissions are required:-dynamodb:BatchGetItem-dynamodb:BatchWriteItem-dynamodb:GetItem-dynamodb:PutItem-dynamodb:Query-dynamodb:UpdateItemModelingCreate a Pydantic model specifically for storage. This should generally not be shared
in API contracts or other external interfaces to adhere to single-responsibility principle.frompydanticimportBaseModelfromtypingimportOptionalclassFilmActor(BaseModel):id:strname:strreview:Optional[str]Reserved Attribute NamesThis package implicitly uses the following attributes, and therefore they should not be used in any
object model classes.AttributeUsage_table_item_idDefault Partition Key name, can be overridden in repository instantiation.Not a default in v2_table_content_idDefault Sort Key name, can be overridden in repository instantiation.Not a default in v2_timestampAutomatic ISO-formatted timestamp for when record was created or updated_object_versionInternal object versioning integer, automatically incremented on update, and can be used to enforce conditional updates_ttlOptionally used to store the DynamoDB TTL expiry value, which must be declared in the table'sTimeToLiveSpecificationVersion 1 (Deprecated)The original repository implementation that still exists but will no longer receive updates.
This will be removed in a future release.V1 Documentationhas been moved to keep the main README a reasonable size.Version 2 (Experimental)This is the implementation that is intended for longer-term maintenance and will receive updates.InstantiationV2 removes thebuildconvenience function found in V1 and provides only a__init__constructor. Since V2 is async-first
the table and resource objects need to be constructed fromaioboto3A working example can be found in theintegration tests conftest.pyfromaioboto3importSessionfrompydantic_dynamo.v2.repositoryimportDynamoRepositoryfromtests.modelsimportExample# Use your record model insteadsession=Session()boto3_kwargs={"service_name":"dynamodb"}# endpoint_url, region_name, etc.asyncwithsession.resource(**boto3_kwargs)asresource:repo=DynamoRepository[Example](item_class=Example,partition_prefix="test",partition_name="integration",content_type="example",table_name="table_name",partition_key="PARTITION_KEY",sort_key="SORT_KEY",table=awaitresource.Table("table_name"),resource=resource,consistent_reads=False,# default)The repository class also implementsAsyncContextManagerso it can be readily used inasync with...statements.
This may make it easier to properly manage other object dependencies in factory/builder functions.fromcontextlibimportasynccontextmanagerfromaioboto3importSessionfrompydantic_dynamo.v2.repositoryimportDynamoRepositoryfromtests.modelsimportExample# Use your record model instead@asynccontextmanagerasyncdefbuild_repo()->DynamoRepository[Example]:session=Session()boto3_kwargs={"service_name":"dynamodb"}# endpoint_url, region_name, etc.asyncwithsession.resource(**boto3_kwargs)asresource,DynamoRepository[Example](item_class=Example,partition_prefix="test",partition_name="integration",content_type="example",table_name="table_name",partition_key="PARTITION_KEY",sort_key="SORT_KEY",table=awaitresource.Table("table_name"),resource=resource,consistent_reads=False,# default)asrepo:yieldrepoThere is also a synchronous variant that can be used if you don't want to work with async/await and async generators
in your business code. Most of the subsequent documentation is focused on the async variant.fromaioboto3importSessionfrompydantic_dynamo.v2.repositoryimportDynamoRepositoryfrompydantic_dynamo.v2.sync_repositoryimportSyncDynamoRepositoryfromtests.modelsimportExample# Use your record model insteadsession=Session()boto3_kwargs={"service_name":"dynamodb"}# endpoint_url, region_name, etc.asyncwithsession.resource(**boto3_kwargs)asresource:repo=DynamoRepository[Example](item_class=Example,partition_prefix="test",partition_name="integration",content_type="example",table_name="table_name",partition_key="PARTITION_KEY",sort_key="SORT_KEY",table=awaitresource.Table("table_name"),resource=resource,consistent_reads=False,# default)sync_repo=SyncDynamoRepository[Example](async_repo=repo)Consistent ReadsBy default, both boto3 and this library use eventually consistent read operations.
You can configure a repository instance to use strongly consistent reads by passing theconsistent_readskwarg asTrueduring instantiation.You canread more about read consistency in AWS's documentation.Saving DataData is saved using an instance of the genericPartitionedContent[ObjT]class found inmodels.py. Thepartition_idsandcontent_idsareList[str].
Each value in the list is concatenated before saving, and prefixed with the repository's configured values.For thecontent_idsfield, you can leverage this to achieve degrees of query-ability for
more complex use cases, eg:content_ids=["usa", "ny", "saratoga", "12020"]will result in a
sort key value ofusa#ny#saratoga#12020that can be efficiently queried with DynamoDB'sbegins_withcondition, utilized in this library'slistfunction.It's wise to ensure that any values being used in the partition and content IDs are also retained as
fields on the model object as well, which will make updates easier to perform.This is less critical now
since the V2 responses include reifiedPartitionedContent[ObjT]objects instead of just the modeled records in V1.Put Single ItemThis leverages the DynamoDB Put operation, and will overwrite an existing item with identical partition and content IDs.frompydantic_dynamo.modelsimportPartitionedContentfromuuidimportuuid4id1=str(uuid4())actor1=FilmActor(id=id1,name="Daniel Day-Lewis")awaitrepo.put(PartitionedContent[FilmActor](partition_ids=[],content_ids=[id1],item=actor1))Put Multiple ItemsWhen saving more than one item, you can use a batch operation that will utilize DynamoDB'swrite_batchoperation, which will more efficiently buffer data and minimize the total number of network calls compared to callingputin a loop.frompydantic_dynamo.modelsimportPartitionedContentfromuuidimportuuid4id1=str(uuid4())actor1=FilmActor(id=id1,name="Daniel Day-Lewis")id2=str(uuid4())actor2=FilmActor(id=id2,name="Steve Buscemi")awaitrepo.put_batch((PartitionedContent[FilmActor](partition_ids=[],content_ids=[id1],item=actor1),PartitionedContent[FilmActor](partition_ids=[],content_ids=[id2],item=actor2),))Update an itemNB: Please review the limitation inissue #1Updates are handled in a somewhat more complex and manual manner using anUpdateCommandobject.
Since this is constructed by sendingDict[str, Any], dictionary entries are validated against
the pydantic model's schema before sending data to DynamoDB.set_commandscan be used to map attributes' names to a new value.increment_attrscan be used to increment attributes' current values by some integer.append_attrscan be used to extend aListattribute's valuescurrent_versioncan be used to enforce a check on the object's version number to
adhere to an object versioning pattern.frompydantic_dynamo.modelsimportUpdateCommandawaitrepo.update(partition_id=None,content_id=[id1],command=UpdateCommand(set_commands={"review":"Talented, but unfriendly in Gangs of New York"}))Reading DataGet ItemFinally, something simple to document. This gets a single item by its partition and content IDs,
returning aGetResponse[FilmActor]with acontentvalue ofNoneif no item is found.This example would retrieve just the first actor item.fromtypingimportOptionalfrompydantic_dynamo.modelsimportPartitionedContentfrompydantic_dynamo.v2.modelsimportGetResponseresponse:GetResponse=awaitrepo.get(partition_id=None,content_id=[id1])item:Optional[PartitionedContent[FilmActor]]=response.contentReading Multiple ItemsThis and all subsequent read operations returnAsyncIterable[BatchResponse[ObjT]]references. These cannot be directly awaited,
but rather need to be iterated in an async loop. SeePEP-525for details and this
repository's tests for complete examples.Most DynamoDB batch operations implement a continuation token pattern, where the API will return a subset of the data
that matches the parameters up to some stored size (eg. 1MB) and then includes a continuation token to use in a subsequent
API call.A singleBatchResponse[ObjT]represents one API response from DynamoDB, and contains anIterable[ObjT]reference. This leaves
you as the consumer in complete control whether you should lazily stream the records or project them all into a list.Get Batch by KeyThis leverages DynamoDB'sbatch_get_itemAPI to collect multiple items by their partition and content IDs.
This is often useful after having collected a previous set of records that have potentially related
items that you want to retrieve, and then associate the two in a subsequent mapping logic layer.This example would retrieve both actor items in a single network request.fromtypingimportListfrompydantic_dynamo.modelsimportPartitionedContentitems:List[PartitionedContent[FilmActor]]=[contentasyncforresponseinrepo.get_batch([(None,[id1]),(None,[id2])])forcontentinresponse.contents]Listing ItemsThe following two functions leverage DynamoDB'squeryAPI and offers the ability
to filter on partial content ID values, change sort order, limit the quantity of items.These operations support optional kwargs:limitinteger to cap the number of records returnedsort_ascendingboolean to change sort order based on the sort key value, which defaults toTrueFilteringYou may also pass an optionalFilterCommandto filter on non-key attributes. All fields
on this object are optional, and are applied utilizingandlogic.frompydantic_dynamo.modelsimportFilterCommand# Find actors without a `review` attributefilter1=FilterCommand(not_exists={"review"})# Find actors who are talented but unfriendly in Gangs of New Yorkfilter2=FilterCommand(equals={"review":"Talented, but unfriendly in Gangs of New York"})# Find actors who are not talented but unfriendly in Gangs of New Yorkfilter3=FilterCommand(not_equals={"review":"Talented, but unfriendly in Gangs of New York"})These operations are more interesting with a more complex data model, so let's pretend there might be
more than one opinion on a given actor, so we'll store many reviews instead of just my own.Let's define a review model that can be related to an actor:frompydanticimportBaseModelfromdatetimeimportdatetimeclassActorReview(BaseModel):id:stractor_id:strcreated:datetimereview:strand then pretend we save a few of them:fromdatetimeimportdatetime,timezonefromuuidimportuuid4now=datetime.now(tz=timezone.utc)awaitrepo.put_batch((PartitionedContent[ActorReview](partition_ids=[actor1.id],content_ids=[now.isoformat()],item=ActorReview(id=str(uuid4()),actor_id=actor1.id,created=now,review="I really thought he was the hero of this movie")),PartitionedContent[ActorReview](partition_ids=[actor1.id],content_ids=[now.isoformat()],item=ActorReview(id=str(uuid4()),actor_id=actor1.id,created=now,review="He really embodies the New York state of mind"))))ListThis function supports filter items with abegins_withfilter on their content IDs.This example would retrieve all review items foractor1that we previously saved.fromtypingimportListfrompydantic_dynamo.modelsimportPartitionedContentitems:List[PartitionedContent[ActorReview]]=[contentasyncforresponseinrepo.list(partition_id=[actor1.id],content_prefix=None,sort_ascending=True,limit=None,filters=None)forcontentinresponse.contents]This example would retrieve all of the reviews created on a given year, because ISO-formatted datetime
values are lexicographically sortable.fromtypingimportListfrompydantic_dynamo.modelsimportPartitionedContentitems:List[PartitionedContent[ActorReview]]=[contentasyncforresponseinrepo.list(partition_id=[actor1.id],content_prefix=["2023"],sort_ascending=True,limit=None,filters=None)forcontentinresponse.contents]List BetweenThis function supports filter items with abetweenfilter on their content IDs.NB: Ifcontent_start == content_endthis will revert to callinglistusingbegins_with. This prevents
unexpected behavior of returning no records due to how Dynamo's Query API handlesbetweenconditions.This example would retrieve all reviews for actor1 that were created in or after January 2023andin or before March 2023.fromtypingimportListfrompydantic_dynamo.modelsimportPartitionedContentitems:List[PartitionedContent[[FilmActor]]=[contentasyncforresponseinrepo.list_between(partition_id=[actor1.id],content_start=["2023-01"],content_end=["2023-04"],sort_ascending=True,limit=None,filters=None)forcontentinresponse.contents]Write Once RepositoryThis utility grew out of a need to minimize duplicate data being saved to the database, because new
was going to be broadcast to data warehouse for wider analysis.
This wrapper around the v2DynamoRepositorywill return aList[PartitionedContent[ObjT]]for all records actually written to the database. The database will be queried, and each item
will be compared to an existing record, if one exists, using Python equivalency.Assuming the underlying repository is using strongly consistent reads, this should minimize the
number of duplicate writes, although there is no statistical evaluation to prove its efficacy.A note on costs:Cost optimization is not the goal of this feature, but rather achieving a reasonable likelihood of
saving any given record only once.
This actually queries the database to compare input data to the existing data. This may cost more
than just blindly putting the content. However, since write units cost significantly more than read units
(maybe 4x? the pricing is confusing), if you have a high likelihood of duplicate data being saved, thismayactually be able to save you some money. As with all things MIT licensed... no guarantees :)frompydantic_dynamo.v2.write_onceimportWriteOnceRepositorywrite_once=WriteOnceRepository[Example](async_repo=repo)count=10000data=[PartitionedContent(...)for_inrange(count)]actually_written:List[PartitionedContent[ObjT]]=awaitwrite_once.write(data)assertlen(actually_written)==countactually_written_again:List[PartitionedContent[ObjT]]=awaitwrite_once.write(data)assertlen(actually_written_again)==0