Project: YugabyteDB (App Definition and Development / Database)
File: 3-anonymous_dumps.md

In many situation, what we want is simply to export the anonymized data into another database (for testing or to produce statistics). This is what pg_dump_anon does! Paul has a website and a comment section where customers can express their views. He hired a web agency to develop a new design for his website. The agency asked for a SQL export (dump) of the current website database. Paul wants to \"clean\" the database export and remove any personnal information contained in the comment section. Extract the anonymized data from the database Write a custom masking function to handle a JSON field. ``` run-postgres DROP TABLE IF EXISTS website_comment CASCADE; CREATE TABLE website_comment ( id SERIAL PRIMARY KEY, message JSONB ); ``` ```bash curl -Ls https://dali.bo/websitecomment -o /tmp/websitecomment.tsv head /tmp/website_comment.tsv ``` ``` run-postgres COPY website_comment FROM '/tmp/website_comment.tsv' ``` ``` run-postgres SELECT message->'meta'->'name' AS name, message->'content' AS content FROM website_comment ORDER BY id ASC ``` ``` run-postgres CREATE EXTENSION IF NOT EXISTS anon CASCADE; SELECT anon.init(); SELECT setseed(0); ``` The \"comment\" field is filled with personal information and the fact the field does not have a standard schema makes our tasks harder. In general, unstructured data are difficult to mask. As we can see, web visitors can write any kind of information in the comment section. Our best option is to remove this key entirely because there\'s no way to extract personnal data properly. We can clean the comment column simply by removing the \"content\" key! ``` run-postgres SELECT message - ARRAY['content'] FROM website_comment WHERE id=1; ``` First let\'s create a dedicated schema and declare it as trusted. This means the \"anon\" extension will accept the functions located in this schema as valid masking functions. Only a superuser should be able to add functions in this

``` run-postgres CREATE SCHEMA IF NOT EXISTS my_masks; SECURITY LABEL FOR anon ON SCHEMA my_masks IS 'TRUSTED'; ``` Now we can write a function that remove the message content: ``` run-postgres CREATE OR REPLACE FUNCTION mymasks.removecontent(j JSONB) RETURNS JSONB AS $func$ SELECT j - ARRAY['content'] $func$ LANGUAGE SQL ; ``` Let\'s try it! ``` run-postgres SELECT mymasks.removecontent(message) FROM website_comment ``` And now we can use it in a masking rule: ``` run-postgres SECURITY LABEL FOR anon ON COLUMN website_comment.message IS 'MASKED WITH FUNCTION mymasks.removecontent(message)'; ``` Finally we can export an anonymous dump of the table with `pgdumpanon`: ``` bash export PATH=$PATH:$(pg_config --bindir) pgdumpanon --help ``` ``` bash export PATH=$PATH:$(pg_config --bindir) export PGHOST=localhost export PGUSER=paul pgdumpanon boutique --table=website_comment > /tmp/dump.sql ``` Create a database named \"boutique_anon\" and transfer the entire database into it. Pierre plans to extract general information from the metadata. For instance, he wants to calculate the number of unique visitors based on the different IP adresses. But an IP adress is an indirect identifier, so Paul needs to anonymize this field while maintaining the fact that some values appear multiple times. Replace the `remove_content` function with a better one called `clean_comment` that will: Remove the content key Replace the \"name\" value with a fake last name Replace the \"ip_address\" value with its MD5 signature Nullify the \"email\" key HINT: Look at the `jsonbset()` and `jsonbbuild_object()` functions ``` bash export PATH=$PATH:$(pg_config --bindir) export PGHOST=localhost export PGUSER=paul dropdb --if-exists boutique_anon createdb boutique_anon --owner paul pgdumpanon boutique | psql --quiet boutique_anon ``` ``` bash export PGHOST=localhost export PGUSER=paul psql boutique_anon -c 'SELECT COUNT(*) FROM company' ``` ```run-postgres CREATE OR REPLACE FUNCTION mymasks.cleancomment(message JSONB) RETURNS JSONB VOLATILE LANGUAGE SQL AS $func$ SELECT jsonb_set( message, ARRAY['meta'], jsonbbuildobject( 'name',anon.fakelastname(), 'ipaddress', md5((message->'meta'->'ipaddr')::TEXT), 'email', NULL ) ) - ARRAY['content']; $func$; ``` ``` run-postgres SELECT mymasks.cleancomment(message) FROM website_comment; ``` ``` run-postgres SECURITY LABEL FOR anon ON COLUMN website_comment.message IS 'MASKED WITH FUNCTION mymasks.cleancomment(message)'; ```