Filtering a Python Dict through Flattening
By: Shane R. Spencer
Tags: Python, Dictionary Comprehension, Programming, Functional Programming, Coding

As we move to document models in our databases it is important to have good tools allowing us to adequately filter out unnecessary and sensitive information.

Photo by Jessica Ruscello on Unsplash

For this article we will be using the flatten-dict library available at https://github.com/ianlini/flatten-dict and installable via pip install flatten-dict .

Example document that needs to be filtered:

user = {

"name": "Bob Robertson",

"classification": {

"kingdom": "animalia",

"phylum": "chordata",

"class": "mammalia",

"order": "primates",

"suborder": "haplorhini",

"infraorder": "simiiformes",

"family": "hominidae",

"subfamily": "homininae",

"tribe": "hominini",

"genus": "homo",

"species": "h. sapiens",

},

"statistics": {

"dimensions": {

"height_cm": 194.2,

"weight_kg": 96.52,

},

"digits": {

"toes": 10,

"fingers": 10,

}

},

}

Let’s pretend that a lot of this information is used by the backend service but not used by the dashboard representing the user. We will want to filter out all but the bold contents: name, classification species and only include dimensions from statistics.

First thing we should do is flatten the dictionary to a dot notated list. This assumes that all keys in the document will not already contain a dot ( . ).

from flatten_dict import flatten flattened_user_doc = flatten(user_doc, reducer="dot")

The resulting contents of flattened_user_doc is:

{

"name": "Bob Robertson",

"classification.kingdom": "animalia",

"classification.phylum": "chordata",

"classification.class": "mammalia",

"classification.order": "primates",

"classification.suborder": "haplorhini",

"classification.infraorder": "simiiformes",

"classification.family": "hominidae",

"classification.subfamily": "homininae",

"classification.tribe": "hominini",

"classification.genus": "homo",

"classification.species": "h. sapiens",

"statistics.dimensions.height_cm": 194.2,

"statistics.dimensions.weight_kg": 96.52,

"statistics.digits.toes": 10,

"statistics.digits.fingers": 10

}

OK. Simple enough. Now lets use regular expressions and a list of matchers to act as a filter and then unflatten the filtered result.

import re filters = [

re.compile("^name$"),

re.compile("^classification.species$"),

re.compile("^statistics.dimensions"),

] filtered_flattened_user_doc = {

k: v

for k, v in flattened_user_doc.items()

if [True for f in filters if f.match(k)]

} from flatten_dict import unflatten filtered_user_doc = unflatten(filtered_flattened_user_doc, splitter="dot")

The resulting contents of filtered_flattened_user_doc is now:

{

"name": "Bob Robertson",

"classification.species": "h. sapiens",

"statistics.dimensions.height_cm": 194.2,

"statistics.dimensions.weight_kg": 96.52

}

And finally we have filtered_user_doc :

{

"name": "Bob Robertson",

"classification": {

"species": "h. sapiens"

},

"statistics": {

"dimensions": {

"height_cm": 194.2,

"weight_kg": 96.52

}

}

}

I believe there are some other modules out there in the wild that help perform this filtering. Using flatten and unflatten from the flatten-dict module and the built in regular expression is straight forward and simple.

You don’t have to use regular expressions. I used them so that I can match partial dot notated paths.

You don’t have to use dots. You can use tuples with flatten_dict and a list of tuples to match against as well.

I’m currently using this code to filter down combined documents from account, user, and user device information pulled from various sources. Using explicit filtering allows me to worry a little less about what is being transmitted to the user as I make document and model changes during the development and ongoing maintenance of my project.