Package: microdata-tools
microdata-toolsTools for themicrodata.noplatformInstallationmicrodata-toolscan be installed from PyPI using pip:pip install microdata-toolsUsageOnce you have your metadata and data files ready to go, they should be named and stored like this:my-input-directory/
    MY_DATASET_NAME/
        MY_DATASET_NAME.csv
        MY_DATASET_NAME.jsonThe CSV file is optional in some cases.Package datasetThepackage_dataset()function will encrypt and package your dataset as a tar archive. The process is as follows:Generate the symmetric key for a dataset.Encrypt the dataset data (CSV) using the symmetric key and store the encrypted file as<DATASET_NAME>.csv.encrEncrypt the symmetric key using the asymmetric RSA public keymicrodata_public_key.pemand store the encrypted file as<DATASET_NAME>.symkey.encrGather the encrypted CSV, encrypted symmetric key and metadata (JSON) file in one tar file.Unpackage datasetTheunpackage_dataset()function will untar and decrypt your dataset using themicrodata_private_key.pemRSA private key.The packaged file has to have the<DATASET_NAME>.tarextension. Its contents should be as follows:<DATASET_NAME>.json: Required medata file.<DATASET_NAME>.csv.encr: Optional encrypted dataset file.<DATASET_NAME>.symkey.encr: Optional encrypted file containing the symmetrical key used to decrypt the dataset file. Required if the.csv.encrfile is present.Decryption uses the RSA private key located atRSA_KEY_DIR.The packaged file is then stored inoutput_dir/archive/unpackagedafter a successful run oroutput_dir/archive/failedafter an unsuccessful run.ExamplePython script that uses a RSA public key namedmicrodata_public_key.pemand packages a dataset:frompathlibimportPathfrommicrodata_toolsimportpackage_datasetRSA_KEYS_DIRECTORY=Path("tests/resources/rsa_keys")DATASET_DIRECTORY=Path("tests/resources/input_package/DATASET_1")OUTPUT_DIRECTORY=Path("tests/resources/output")package_dataset(rsa_keys_dir=RSA_KEYS_DIRECTORY,dataset_dir=DATASET_DIRECTORY,output_dir=OUTPUT_DIRECTORY,)ValidationOnce you have your metadata and data files ready to go, they should be named and stored like this:my-input-directory/
    MY_DATASET_NAME/
        MY_DATASET_NAME.csv
        MY_DATASET_NAME.jsonNote that the filename only allows upper case letters A-Z, number 0-9 and underscores.Import microdata-tools in your script and validate your files:frommicrodata_toolsimportvalidate_datasetvalidation_errors=validate_dataset("MY_DATASET_NAME",input_directory="path/to/my-input-directory")ifnotvalidation_errors:print("My dataset is valid")else:print("Dataset is invalid :(")# You can print your errors like this:forerrorinvalidation_errors:print(error)For a more in-depth explanation of usage visitthe usage documentation.Data format descriptionA dataset as defined in microdata consists of one data file, and one metadata file.The data file is a csv file seperated by semicolons. A valid example would be:000000000000001;123;2020-01-01;2020-12-31;
000000000000002;123;2020-01-01;2020-12-31;
000000000000003;123;2020-01-01;2020-12-31;
000000000000004;123;2020-01-01;2020-12-31;Read more about the data format and columns inthe documentation.The metadata files should be in json format. The requirements for the metadata is best described through thePydantic model,the examples, andthe documentation.