Package: dynamodb-csv
DynamoDB CSV utilityA utility that allows CSV import / export to DynamoDB on the command lineGive a ⭐️ if you like this tool!IntroductionI made this command because I didn't have any tools to satisfy my modest desire to make it easy to import CSV files into DynamoDB.
Written in a simple Python script, it's easy to parse and modify.It works for me.Getting started 🚀Install$pipinstalldynamodb-csv
$dynamodb-csv-h
usage:dynamodb-csv[-h][-v][-i][-e][--truncate][--move]-t[TABLE...][-idxINDEX][-fFILE][-oOUTPUT][--ignore][--profilePROFILE]ImportCSVfileintoDynamoDBtableutilities

optionalarguments:-h,--helpshowthishelpmessageandexit-v,--versionshowversion-i,--impmodeimport-e,--expmodeexport--truncatemodetruncate--movemodemove-t[TABLE...],--table[TABLE...]DynamoDBtablename-idxINDEX,--indexINDEXDynamoDBindexname-fFILE,--fileFILEUTF-8CSVfilepathrequiredimportmode-oOUTPUT,--outputOUTPUToutputfilepathrequiredexportmode--ignoreignoreimporterror--profilePROFILEusingAWSprofileInstall for developer$python-mvenvvenv
$.venv/bin/activate
$pythonsetup.pyinstall
$dynamodb-csv-hOr$python-mvenvvenv
$.venv/bin/activate
$pipinstall-rrequirements-dev.txt
$exportPYTHONPATH=`pwd`$pythonapp/main.py-hFor Windows>python-mvenvvenv
>venv\Scripts\activate
>pipinstall-rrequirements-dev.txt
>setPYTHONPATH=%cd%
>pythonapp/main.py-hOr you can use devcontainer.Use Docker image$dockerrun--rm-v${PWD}/:/localdanishi/dynamodb-csv:tagname-i-tmy_table-fsample.csvFor Windows>dockerrun--rm-v%cd%/:/localdanishi/dynamodb-csv:tagname-i-tmy_table-fsample.csvOrGitHub PackagesCreate your config.ini file on current directory[AWS]AWS_ACCESS_KEY_ID=your_access_keyAWS_SECRET_ACCESS_KEY=your_secret_keyREGION=your_dynamodb_table_region# Option#ENDPOINT_URL=http://dynamodb-local:8000Not required if AWS profile is specified as a parameter.Create your CSV and CSV spec file[!NOTE]
Prepare a UTF-8 CSV file of the format you want to import into your DynamoDB table and a file that defines that format.For examplePlease refer to this writing method.sample.csvStringPK,NumberSK,DecimalValue,BooleanValue,NullValue,JsonValue,StringListValues,DecimalListValues
foo,1,1.23,TRUE,,"[{""string"" : ""value""},{""number"" : 100}]",foo bar baz,10 10.1 20
foo,2,0.001,,,"[{""boolean"" : true}]",リンゴ バナナ スイカ,10 10.1 20
foo,3,1,,,"[{""boolean"" : false}]",,sample.csv.spec# sample.csv data format specification# String : S# Integer : I# Decimal : D# Boolean : B (blank false)# Json : J# StringList : SL# StringSet : SS# DecimalList : DL# DecimalSet : DS[CSV_SPEC]StringPK=SNumberSK=IDecimalValue=DBooleanValue=BNullValue=SJsonValue=JStringListValues=SLStringSetValues=SSDecimalListValues=DLDecimalSetValues=DSCreate DynamoDB table[!NOTE]
You need to have created a DynamoDB table that meets your specifications.$awsdynamodbcreate-table--cli-input-jsonfile://my_table.json--regionap-northeast-1
$awsdynamodbdescribe-table--table-namemy_table{"Table":{"AttributeDefinitions":[{"AttributeName":"NumberSK","AttributeType":"N"},{"AttributeName":"StringPK","AttributeType":"S"}],"TableName":"my_table","KeySchema":[{"AttributeName":"StringPK","KeyType":"HASH"},{"AttributeName":"NumberSK","KeyType":"RANGE"}],"TableStatus":"ACTIVE","CreationDateTime":"2022-06-26T21:19:21.767000+09:00","ProvisionedThroughput":{"NumberOfDecreasesToday":0,"ReadCapacityUnits":5,"WriteCapacityUnits":5},"TableSizeBytes":0,"ItemCount":0,"TableArn":"arn:aws:dynamodb:ap-northeast-1:XXXXXXXXXXX:table/my_table","TableId":"XXXXXXXX-925b-4cb1-8e3a-604158118c3f","GlobalSecondaryIndexes":[{"IndexName":"NumberSK-index","KeySchema":[{"AttributeName":"NumberSK","KeyType":"HASH"}],"Projection":{"ProjectionType":"INCLUDE","NonKeyAttributes":["DecimalValue","JsonValue"]},"IndexStatus":"ACTIVE","ProvisionedThroughput":{"NumberOfDecreasesToday":0,"ReadCapacityUnits":5,"WriteCapacityUnits":5},"IndexSizeBytes":0,"ItemCount":0,"IndexArn":"arn:aws:dynamodb:ap-northeast-1:XXXXXXXXXXX:table/my_table/index/NumberSK-index"}]}}CSV import into TableThis command requires a CSV spec file in the same directory.$dynamodb-csv-i-tmy_table-fsample.csv
pleasewaitmy_tableimportingsample.csv
300it[00:00,19983.03it/s]100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|300/300[00:07<00:00,40.97it/s]my_tablecsvimported300itemsIt is processed at high speed by batch write.Ignore optionIf there is an error such as a key schema mismatch, you can give the option to ignore the CSV record.$dynamodb-csv-i-tmy_table-fsample.csv--ignore
pleasewaitmy_tableimportingsample.csv
300it[00:00,19983.03it/s]100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|300/300[00:07<00:00,40.97it/s]my_tablecsvimported299itemsand1erroritemsNo batch write is done when this option is used.Import optionsBy default, if CSV has an empty value, it will be set to empty.There are options to convert this to Null or not to set the attribute itself.[IMPORT_OPTION]ConvertBlankToNullAttrs=NullValue,JsonValueConvertBlankToDropAttrs=DecimalValueExport table to CSVYou will also need to expand the same data to multiple tables.Therefore, data can be exported.As with import, you need a CSV spec file.$dynamodb-csv-e-tmy_table-osample_exp.csv
pleasewaitmy_tableexportingsample_exp.csv100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|300/300[00:00<00:00,16666.77it/s]my_tablecsvexported300itemsUse index$dynamodb-csv-e-tmy_table-idxNumberSK-index-osample_gsi_exp.csvUse Query$dynamodb-csv-e-tmy_table-idxNumberSK-index-osample_query_exp.csv# sample_query_exp.csv data format specification# Integer : I# String : S# Decimal : D# Json : J[QUERY_OPTION]PKAttribute=NumberSKPKAttributeValue=1PKAttributeType=I[CSV_SPEC]NumberSK=IStringPK=SDecimalValue=DJsonValue=JQuery optionskeydescriptionexamplePKAttributePartition key attribute namePKAttributeValuePartition key attribute query valuePKAttributeTypePartition key attribute data typeSKAttributeSort key attribute nameSKAttributeValuesSort key attribute query value or valuesex.fooorfoo,barSKAttributeTypeSort key attribute data typeSKAttributeExpressionSort key attribute query expressionex.begins_withbetweeneqgtgteltlte$dynamodb-csv-e-tmy_table-osample_query_exp2.csv[QUERY_OPTION]PKAttribute=StringPKPKAttributeValue=barPKAttributeType=SSKAttribute=NumberSKSKAttributeValues=50,100SKAttributeType=ISKAttributeExpression=betweenTable truncateAlso, since you may want to erase unnecessary data during the import experiment, we have prepared a command to discard it.$dynamodb-csv--truncate-tmy_table
my_tablescan300items
pleasewaitmy_tabletruncating100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|300/300[00:07<00:00,40.95it/s]my_tabletruncated[!CAUTION]
This operation is irreversible. Take care.Table moveMove all items from table to table.
A table with the same schema must be prepared in advance.Table items is not deleted and behaves like a copy.$dynamodb-csv--move-tmy_table_frommy_table_to
my_table_fromscan300items
pleasewaitmy_table_tomoving100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|300/300[00:15<00:00,20.00it/s]my_table_tomoved300itemsLicenseSeeLICENSESpecial ThanksCode contributors 🤝AppendixUser guideUser guide (for japanese)