Package: dynamo-pandas
dynamo-pandasMake working with pandas data and AWS DynamoDB easy.MotivationThis package aims a making the transfer of data between pandas dataframes and DynamoDB as simple as possible. To meet this goal, the package offers two key features:Automatic conversion of pandas data types to DynamoDB supported data types.A simple, high level interface toputdata from a dataframe into a DynamoDB table andgetall or selected items from a table into a dataframe.DocumentationThe project's documentation is available athttps://dynamo-pandas.readthedocs.io/.Requirementspython>=3.7pandas>=1boto3Installationpython -m pip install dynamo-pandasThis will install the package and its dependencies except forboto3which is not installed by default to avoid unnecessary installation when building Lambda layers.To includeboto3as part of the installation, add theboto3"extra" this way:python -m pip install dynamo-pandas[boto3]Example UsageConsider the pandas DataFrame below.>>>print(players_df)player_idlast_playplay_timeratingbonus_points0player_one2021-01-1822:47:232days17:41:554.331player_two2021-01-1919:07:540days22:07:343.812player_three2021-01-2110:22:431days14:01:192.543player_four2021-01-2213:51:120days03:45:494.8<NA>The columns of the dataframe use different data types, some of which are not natively supported by DynamoDB, like numpy.datetime64, timedelta64 and pandas' nullable integers.>>>players_df.info()<class'pandas.core.frame.DataFrame'>RangeIndex:4entries,0to3Datacolumns(total5columns):#   Column        Non-Null Count  Dtype----------------------------0player_id4non-nullobject1last_play4non-nulldatetime64[ns]2play_time4non-nulltimedelta64[ns]3rating4non-nullfloat644bonus_points3non-nullInt8dtypes:Int8(1),datetime64[ns](1),float64(1),object(1),timedelta64[ns](1)memoryusage:264.0+bytesStoring the rows of this dataframe to DynamoDB requires multiple data type conversions.>>>fromdynamo_pandasimportput_df,get_df,keysTheput_dffunction adds or updates the rows of a dataframe into the specified table, taking care of the required type conversions (the table must be already created and the primary key column(s) be present in the dataframe).>>>put_df(players_df,table="players")Theget_dffunction retrieves the items matching the speficied key(s) from the table into a dataframe.>>>df=get_df(table="players",keys=[{"player_id":"player_three"},{"player_id":"player_one"}])>>>print(df)bonus_pointsplayer_idlast_playratingplay_time04player_three2021-01-2110:22:432.51days14:01:1913player_one2021-01-1822:47:234.32days17:41:55In the case where only a partition key is used, thekeysfunction simplifies the generation of the keys list.>>>df=get_df(table="players",keys=keys(player_id=["player_two","player_four"]))>>>print(df)bonus_pointsplayer_idlast_playratingplay_time01.0player_two2021-01-1919:07:543.80days22:07:341NaNplayer_four2021-01-2213:51:124.80days03:45:49The data types returned by theget_dffunction are basic types and no automatic type conversion is attempted.>>>df.info()<class'pandas.core.frame.DataFrame'>RangeIndex:2entries,0to1Datacolumns(total5columns):#   Column        Non-Null Count  Dtype----------------------------0bonus_points1non-nullfloat641player_id2non-nullobject2last_play2non-nullobject3rating2non-nullfloat644play_time2non-nullobjectdtypes:float64(2),object(3)memoryusage:208.0+bytesThedtypeparameter of theget_dffunction allows specifying the desired data types.>>>df=get_df(...table="players",...keys=keys(player_id=["player_two","player_four"]),...dtype={..."bonus_points":"Int8",..."last_play":"datetime64[ns, UTC]",...# "play_time": "timedelta64[ns]"  # See note below....}...)Note: Due to a known bug in pandas, timedelta strings cannot currently be converted back to Timedelta type via this parameter (ref.https://github.com/pandas-dev/pandas/issues/38509). Use the pandas.to_timedelta function instead:>>>df.play_time=pd.to_timedelta(df.play_time)>>>df.info()<class'pandas.core.frame.DataFrame'>RangeIndex:2entries,0to1Datacolumns(total5columns):#   Column        Non-Null Count  Dtype----------------------------0bonus_points1non-nullInt81player_id2non-nullobject2last_play2non-nulldatetime64[ns,UTC]3rating2non-nullfloat644play_time2non-nulltimedelta64[ns]dtypes:Int8(1),datetime64[ns,UTC](1),float64(1),object(1),timedelta64[ns](1)memoryusage:196.0+bytesOmitting thekeysparameter performs a scan of the table and returns all the items.>>>df=get_df(table="players")>>>print(df)bonus_pointsplayer_idlast_playratingplay_time04.0player_three2021-01-2110:22:432.51days14:01:191NaNplayer_four2021-01-2213:51:124.80days03:45:4923.0player_one2021-01-1822:47:234.32days17:41:5531.0player_two2021-01-1919:07:543.80days22:07:34LicenseReleased under the terms of theMIT License.