Package: skymantle-mock-data-forge
Skymantle Mock Data ForgeA package for managing test data on aws resources through configuration, when running integration and end-to-end tests in the cloud. Currently the package supports managing data on DynamoDb and S3.InstallationTo install use:pip3 install skymantle_mock_data_forgeUsageUse the forge factory to manage data to multiple destinations (any combination of DynamoDB tables and S3 buckets). An id is used to specify each unique destination. The forge factor provides the following functions:load_data- will load data across all destinations or the destination of the provided forge IDget_data- will return data across all destinations or the data for the provided forge ID, will only return data created by the forgeadd_key- when new data is created through tests you can provide it's key so that it's included in thecleanup_datacallcleanup_data- will remove data across all destinations or the destination of the provided forge IDExamplesmanage data in DynamoDB, assumes aws credentials environment variable are set, such asAWS_PROFILEfromskymantle_mock_data_forge.forge_factoryimportForgeFactoryconfig=[{"forge_id":"some_config_id","dynamodb":{"table":{"name":"some_table"},"primary_key_names":["PK"],"items":[{"data":{"PK":"some_key_1","Description":"Some description 1"}}],},}]factory=ForgeFactory(config)factory.load_data()# perform testsfactory.add_key("some_config_id",{"PK":"some_key_2"})factory.cleanup_data()Load config from a json filefromskymantle_mock_data_forge.forge_factoryimportForgeFactoryfactory=ForgeFactory("path/to/file.json")factory.load_data()# perform testsfactory.add_key("some_config_id",{"PK":"some_key_2"})factory.cleanup_data()Managing multiple destinationsfromskymantle_mock_data_forge.forge_factoryimportForgeFactoryconfig=[{"forge_id":"some_config_id_1","dynamodb":{"table":{"name":"some_table"},"primary_key_names":["PK"],"items":[{"data":{"PK":"some_key_1","Description":"Some description 1"}}],},},{"forge_id":"some_config_id_2","s3":{"bucket":{"name":"some_bucket"},"s3_objects":[{"key":"some_key","data":{"base64":"SGVsbG8gV29ybGQh"}}],},}]factory=ForgeFactory(config)# To load a single destinationfactory.load_data("some_config_id_2")# To load all destinationsfactory.load_data()# ...Provide forge factory with an AWS sessionfromboto3importSession# ...session=Session(profile_name="developer")factory=ForgeFactory(config,session)# ...Use queries to get specific data for use in tests by querying custom tags.fromdatetimeimportUTC,datetimefromskymantle_mock_data_forge.forge_factoryimportForgeFactoryconfig=[{"forge_id":"some_config_id","dynamodb":{"table":{"name":"some_table"},"primary_key_names":["PK"],"items":[{"tags":{"tests":["test_get_items","test_get_item","test_update_item",],},"data":{"PK":"some_key_1","Description":"Some description 1"},},{"tags":{"tests":"test_item_delete"},"data":{"PK":"some_key_2","Description":"Some description 2"},},],},}]factory=ForgeFactory(config,overrides=overrides)factory.load_data()#...# Unit test for deleting an itemdata=factory.get_data(query={"StringEquals":{"tests":"test_item_delete"}})pK_to_delete=data[0]["data"]["PK]#...Override specific values at run time. Useful when config files are stored in static json files or handling datesfromdatetimeimportUTC,datetimefromskymantle_mock_data_forge.forge_factoryimportForgeFactoryfromskymantle_mock_data_forge.modelsimportOverideTypeconfig=[{"forge_id":"some_config_id","dynamodb":{"table":{"name":"some_table"},"primary_key_names":["PK"],"items":[{"data":{"PK":"some_key_1","Description":"Some description 1","CreateDate":""}}],},}]overrides=[{"key_paths":"CreateDate","override_type":OverideType.REPLACE_VALUE,"override":datetime.now(UTC).isoformat(),}]factory=ForgeFactory(config,overrides=overrides)factory.load_data("some_config_id")# ...ConfigurationThe forge factory takes a list of configuration, each item in the lists must represent a single destination, either DynamoDB or S3.For the DynamoDb configuration, the table name can be specific or  provided through an SSM parameter or the output of a CloudFormation stackBy table name{"forge_id":"some_config_id_1","dynamodb":{"table":{"name":"some_table"},"primary_key_names":["PK"],"items":[{"data":{"PK":"some_key_1","Description":"Some description 1"}}],}}By SSM paramter{"forge_id":"some_config_id_1","dynamodb":{"table":{"ssm":"ssm_parameter_key"},"primary_key_names":["PK"],"items":[{"data":{"PK":"some_key_1","Description":"Some description 1"}}],}}By CloudFormation stack output{"forge_id":"some_config_id_1","dynamodb":{"table":{"stack":{"name":"cfn_stack_name","output":"table_output_name"}},"primary_key_names":["PK"],"items":[{"data":{"PK":"some_key_1","Description":"Some description 1"}}],}}For the S3 configuration, the bucket name can be specific or  provided through an SSM parameter or the output of a CloudFormation stack (similar to DynamoDB).The following object data is supported:textjsonbase64csvfile{"forge_id":"some_config_id_1","s3":{"bucket":{"name":"some_bucket"},"s3_objects":[{"key":"some_key_1","data":{"text":"Some Data"}},{"key":"some_key_2","data":{"json":{"some_key":"some_value"}}},{"key":"some_key_3","data":{"base64":"SGVsbG8gV29ybGQh"}},{"key":"some_key_4","data":{"csv":[["a","b"],["c","d"],["e","f"],[1,2],]}},{"key":"some_key_5","data":{"file":"path/to/file"}},]}}Querying Data For TestingCustom tags can be added to data that is managed by the forges, this will make it possible to catagorize and group data for use during tests. Tags are completely optional, but requried for querying.DynamoDB tag example[{"tags":{"tests":["test_get_item","test_update_item"]},"data":{"PK":"some_key_1","Description":"Some description 1"},},{"tags":{"tests":"test_item_delete"},"data":{"PK":"some_key_2","Description":"Some description 2"},},]S3 tag example[{"key":"some_key_1","tags":{"type":"text","tests":["test_1","test_2"],"segment":"reporting"},"data":{"text":"Some Data"},},{"key":"some_key_2","tags":{"type":"json","tests":"test_3"},"data":{"json":{"key":"value"}},}]A query is made up of an operator and 1 or more condition key/value pairs. Supported operators are:StringEquals - the value for the tag must match exactly to one of the items in thes specified tagStringLike - the value must be contained in on of the items in the specified tagIt's also possible to use both operators in the query. When dealing with multiple contitions and/or operators, queries are limited to logicalANDqueries.ExamplesFor the data provided[{"tags":{"record_type":"person","tests":["test_get_items","test_get_item","test_update_item"]},"data":{"PK":"some_key_1","Description":"Some description 1"}},{"tags":{"record_type":"building","tests":["test_get_items","test_get_item","test_update_item"]},"data":{"PK":"some_key_2","Description":"Some description 2"}},{"tags":{"record_type":"building","tests":"item_delete"},"data":{"PK":"some_key_2","Description":"Some description 2"}}]{"StringLike": {"tests": "get"}}will return[{"tags":{"record_type":"person","tests":["test_get_items","test_get_item","test_update_item"]},"data":{"PK":"some_key_1","Description":"Some description 1"}},{"tags":{"record_type":"building","tests":["test_get_items","test_get_item","test_update_item"]},"data":{"PK":"some_key_2","Description":"Some description 2"}}]{"StringEquals": {"tests": "test_get_item", "record_type": "building"}}will return[{"tags":{"record_type":"building","tests":["test_get_items","test_get_item","test_update_item"]},"data":{"PK":"some_key_2","Description":"Some description 2"}}]{"StringEquals": {"record_type": "building"}, "StringLike": {"tests": "test"}}will return[{"tags":{"record_type":"building","tests":["test_get_items","test_get_item","test_update_item"]},"data":{"PK":"some_key_2","Description":"Some description 2"}}]OverridesIn some cases it's easier to store config files in json for sharing. When coming from static sources, overrides allow data to be changed at runtime. this can be used for setting values like the AWS account id for the current context or the current date. The following type of modifications can be used:Replace ValueWill replace the value with a new oneFor example if create date is current "" it can be replaced with "2023-01-01"Format ValueWill used pythonsstr.format(*args)The original value must be a string, the replacement is index based.For example if ddescription is "{} Assemble{}" it back be formated to "Avengers Assemble!"Call Functionwill call a custom function when processing each item.The signature of the funct is func(key: str, value: any, context: dict) -> any:key - The current key (ie: CreateDate)value - The current value for the given keyconext - The current item being builtreturn - the new value.In the case of nested dictionaries, key paths are supported which are "." seperated, key paths will also traverse sub lists. Currently it's not possible to specify an index, all items in the list will be updated.If the same overide is needed for multiple keys then a list of key paths can be used. Currently overrides only work with json structured data.The default behaviour for overrides is to ignore key path errors, however this behaviour can be altered by setting theDATA_FORGE_SUPPRESS_KEY_PATH_ERRORSenvironment variable. Supported values are0,false,noandoff.Example# Current config stored in full or in part in json files[{"forge_id":"some_config_id","dynamodb":{"table":{"name":"some_table"},"primary_key_names":["pk"],"items":[{"data":{"pk":"","description":"Executed on{0}Environment.","audit":{"create_date":"","last_update_date":""},"items":[{"id":""},{"id":""}]}},{"data":{"pk":"","description":"Executed on{0}Environment.","audit":{"create_date":"","last_update_date":""},"items":[{"id":""},{"id":""}]}}],},}]# Overridesdefgenerate_id(key:str,value:any,context:dict)->any:returnstr(uuid.uuid4())current_date=datetime.now(UTC).isoformat()environment=os.environ.get("ENVIRONMENT")overrides=[{"key_paths":["data.pk","data.items.id",],"override_type":OverideType.CALL_FUNCTION,"override":generate_id,},{"key_paths":["data.audit.create_date","data.audit.last_update_date",],"override_type":OverideType.REPLACE_VALUE,"override":current_date,},{"key_paths":"data.description","override_type":OverideType.FORMAT_VALUE,"override":[environment],}]# The resulting data{"data":{"pk":"11184314-b3fd-4a2d-bf79-bb50eabc9985","description":"Executed on Test Environment.","audit":{"create_date":"2024-01-06T22:59:00.469843+00:00","last_update_date":"2024-01-06T22:59:00.469843+00:00"},"items":[{"id":"e61e4001-ac84-43fb-8ba4-0f2ca5972c83"},{"id":"9ea2d4ff-9a01-4ea6-95ea-4f6f01e797ca"}]}},{"data":{"pk":"91aa7686-6dbd-47a1-a779-891436fdfac0","description":"Executed on Test Environment.","audit":{"create_date":"2024-01-06T22:59:00.469843+00:00","last_update_date":"2024-01-06T22:59:00.469843+00:00"},"items":[{"id":"45aef30f-a6da-448b-bbac-2530f5aad558"},{"id":"65722f85-f799-4a85-9415-088bbcd55d8c"}]}},By default, the same list of overrides is distributed to all destinations, if an override needs to vary by destination a forge Id can be provided.overrides=[{# Will go to all destinations"key_paths":"data.text","override_type":OverideType.REPLACE_VALUE,"override":"Some Other Data",},{# Will go to some_config_1"forge_id":"some_config_1","key_paths":"PK","override_type":OverideType.REPLACE_VALUE,"override":"some_other_key_1",},{# Will go to some_config_2"forge_id":"some_config_2","key_paths":"PK","override_type":OverideType.REPLACE_VALUE,"override":"some_other_key_2",}]Source Code Dev NotesThe following project commands are supported:make setup- Installs all dependencies ands creates virtual environmentmake unitTests- runs unit testsmake lintAndAnalysis- Runsruff,banditandblackmake build- Creates distribution