Package: dapla-toolbelt-pseudo
Dapla Toolbelt PseudoPseudonymize, repseudonymize and depseudonymize data on Dapla.FeaturesOther examples can also be viewed through notebook files forpseudoanddepseudoPseudonymizefromdapla_pseudoimportPseudonymizeimportpolarsasplfile_path="data/personer.csv"dtypes={"fnr":pl.Utf8,"fornavn":pl.Utf8,"etternavn":pl.Utf8,"kjonn":pl.Categorical,"fodselsdato":pl.Utf8}df=pl.read_csv(file_path,dtypes=dtypes)# Create DataFrame from file# Example: Single field default encryption (DAEAD)result_df=(Pseudonymize.from_polars(df)# Specify what dataframe to use.on_fields("fornavn")# Select the field to pseudonymize.with_default_encryption()# Select the pseudonymization algorithm to apply.run()# Apply pseudonymization to the selected field.to_polars()# Get the result as a polars dataframe)# Example: Multiple fields default encryption (DAEAD)result_df=(Pseudonymize.from_polars(df)# Specify what dataframe to use.on_fields("fornavn","etternavn")# Select multiple fields to pseudonymize.with_default_encryption()# Select the pseudonymization algorithm to apply.run()# Apply pseudonymization to the selected fields.to_polars()# Get the result as a polars dataframe)# Example: Single field sid mapping and pseudonymization (FPE)result_df=(Pseudonymize.from_polars(df)# Specify what dataframe to use.on_fields("fnr")# Select the field to pseudonymize.with_stable_id()# Map the selected field to stable id.run()# Apply pseudonymization to the selected fields.to_polars()# Get the result as a polars dataframe)The default encryption algorithm is DAEAD (Deterministic Authenticated Encryption with Associated Data). However, if the
field is a valid Norwegian personal identification number (fnr, dnr), the recommended way to pseudonymize is to use
the functionwith_stable_id()to convert the identification number to a stable ID (SID) prior to pseudonymization.
In that case, the pseudonymization algorithm is FPE (Format Preserving Encryption).Note that you may also use a Pandas DataFrame as an input or output, by exchangingfrom_polarswithfrom_pandasandto_polarswithto_pandas. However, Pandas is much less performant, so take special care especially if your
dataset is large.Example:# Example: Single field default encryption (DAEAD)df_pandas=(Pseudonymize.from_pandas(df)# Specify what dataframe to use.on_fields("fornavn")# Select the field to pseudonymize.with_default_encryption()# Select the pseudonymization algorithm to apply.run()# Apply pseudonymization to the selected field.to_pandas()# Get the result as a polars dataframe)Validate SID mappingfromdapla_pseudoimportValidatorimportpolarsasplfile_path="data/personer.csv"dtypes={"fnr":pl.Utf8,"fornavn":pl.Utf8,"etternavn":pl.Utf8,"kjonn":pl.Categorical,"fodselsdato":pl.Utf8}df=pl.read_polars(file_path,dtypes=dtypes)result=(Validator.from_polars(df)# Specify what dataframe to use.on_field("fnr")# Select the field to validate.validate_map_to_stable_id()# Validate that all the field values can be mapped to a SID)# The resulting dataframe contains the field values that didn't have a corresponding SIDresult.to_polars()Asid_snapshot_datecan also be specified to validate that the field values can be mapped to a SID at a specific date:fromdapla_pseudoimportValidatorfromdapla_pseudo.utilsimportconvert_to_dateimportpolarsasplfile_path="data/personer.csv"dtypes={"fnr":pl.Utf8,"fornavn":pl.Utf8,"etternavn":pl.Utf8,"kjonn":pl.Categorical,"fodselsdato":pl.Utf8}df=pl.read_csv(file_path,dtypes=dtypes)result=(Validator.from_polars(df).on_field("fnr").validate_map_to_stable_id(sid_snapshot_date=convert_to_date("2023-08-29")))# Show metadata about the validation (e.g. which version of the SID catalog was used)result.metadata# Show the field values that didn't have a corresponding SIDresult.to_polars()Advanced usagePseudonymizeRead from file systemsfromdapla_pseudoimportPseudonymizefromdaplaimportAuthClientfile_path="data/personer.csv"options={"dtypes":{"fnr":pl.Utf8,"fornavn":pl.Utf8,"etternavn":pl.Utf8,"kjonn":pl.Categorical,"fodselsdato":pl.Utf8}}# Example: Read DataFrame from fileresult_df=(Pseudonymize.from_file(file_path)# Read the data from file.on_fields("fornavn","etternavn")# Select multiple fields to pseudonymize.with_default_encryption()# Select the pseudonymization algorithm to apply.run()# Apply pseudonymization to the selected fields.to_polars(**options)# Get the result as a Pandas DataFrame)# Example: Read dataframe from GCS bucketoptions={"dtypes":{"fnr":pl.Utf8,"fornavn":pl.Utf8,"etternavn":pl.Utf8,"kjonn":pl.Categorical,"fodselsdato":pl.Utf8}}gcs_file_path="gs://ssb-staging-dapla-felles-data-delt/felles/pseudo-examples/andeby_personer.csv"result_df=(Pseudonymize.from_file(gcs_file_path)# Read DataFrame from GCS.on_fields("fornavn","etternavn")# Select multiple fields to pseudonymize.with_default_encryption()# Select the pseudonymization algorithm to apply.run()# Apply pseudonymization to the selected fields.to_polars(**options)# Get the result as a polars dataframe)Pseudonymize using custom keys/keysetsfromdapla_pseudoimportpseudonymize# Pseudonymize fields in a local file using the default key:df=(Pseudonymize.from_polars(df)# Specify what dataframe to use.on_fields("fornavn")# Select the field to pseudonymize.with_default_encryption()# Select the pseudonymization algorithm to apply.run()# Apply pseudonymization to the selected field.to_polars()# Get the result as a polars dataframe)# Pseudonymize fields in a local file, explicitly denoting the key to use:df=(Pseudonymize.from_polars(df)# Specify what dataframe to use.on_fields("fornavn")# Select the field to pseudonymize.with_default_encryption(custom_key="ssb-common-key-2")# Select the pseudonymization algorithm to apply.run()# Apply pseudonymization to the selected field.to_polars()# Get the result as a polars dataframe)pseudonymize(file_path="./data/personer.json",fields=["fnr","fornavn"],key="ssb-common-key-1")# Pseudonymize a local file using a custom keyset:importjsoncustom_keyset=PseudoKeyset(encrypted_keyset="CiQAp91NBhLdknX3j9jF6vwhdyURaqcT9/M/iczV7fLn...8XYFKwxiwMtCzDT6QGzCCCM=",keyset_info={"primaryKeyId":1234567890,"keyInfo":[{"typeUrl":"type.googleapis.com/google.crypto.tink.AesSivKey","status":"ENABLED","keyId":1234567890,"outputPrefixType":"TINK",}],},kek_uri="gcp-kms://projects/some-project-id/locations/europe-north1/keyRings/some-keyring/cryptoKeys/some-kek-1",)df=(Pseudonymize.from_polars(df).on_fields("fornavn").with_default_encryption(custom_key="1234567890")# Note that the custom key has to be the same as "primaryKeyId" in the custom keyset.run(custom_keyset=custom_keyset).to_polars())DepseudonymizeThe "Depseudonymize" functions are almost exactly the same as when pseudonymizing.
The only difference being the lack of a "with_stable_id()"-function.
This is to say, that you cannot map from Stable IDback toFNR as of Jan 2023.fromdapla_pseudoimportDepseudonymizeimportpolarsasplfile_path="data/personer_pseudonymized.csv"dtypes={"fnr":pl.Utf8,"fornavn":pl.Utf8,"etternavn":pl.Utf8,"kjonn":pl.Categorical,"fodselsdato":pl.Utf8}df=pl.read_csv(file_path,dtypes=dtypes)# Create DataFrame from file# Example: Single field default encryption (DAEAD)result_df=(Depseudonymize.from_polars(df)# Specify what dataframe to use.on_fields("fornavn")# Select the field to depseudonymize.with_default_encryption()# Select the depseudonymization algorithm to apply.run()# Apply depseudonymization to the selected field.to_polars()# Get the result as a polars dataframe)# Example: Multiple fields default encryption (DAEAD)result_df=(Depseudonymize.from_polars(df)# Specify what dataframe to use.on_fields("fornavn","etternavn")# Select multiple fields to depseudonymize.with_default_encryption()# Select the depseudonymization algorithm to apply.run()# Apply depseudonymization to the selected fields.to_polars()# Get the result as a polars dataframe)Repseudonymize## TODONote that depseudonymization requires elevated access privileges.RequirementsPython >= 3.10Dependencies can be found inpyproject.tomlInstallationYou can installDapla Toolbelt PseudoviapipfromPyPI:pip install dapla-toolbelt-pseudoUsagePlease see theReference Guidefor details.ContributingContributions are very welcome.
To learn more, see theContributor Guide.LicenseDistributed under the terms of theMIT license,Dapla Toolbelt Pseudois free and open source software.IssuesIf you encounter any problems,
pleasefile an issuealong with a detailed description.CreditsThis project was generated fromStatistics Norway'sSSB PyPI Template.