Package: django-scrubber
Django Scrubberdjango_scrubberis a django app meant to help you anonymize your project's database data. It destructively alters data
directly on the DB and thereforeshould not be used on production.The main use case is providing developers with realistic data to use during development, without having to distribute
your customers' or users' potentially sensitive information.
To accomplish this,django_scrubbershould be plugged in a step during the creation of your database dumps.Simply mark the fields you want to anonymize and call thescrub_datamanagement command. Data will be replaced based
on differentscrubbers(see below), which define how the anonymous content will be generated.If you want to be sure that you don't forget any fields in the ongoing development progress, you can use the
management commandscrub_validationin your CI/CD pipeline to check for any missing fields.InstallationSimply run:pip install django-scrubberAnd adddjango_scrubberto your djangoINSTALLED_APPS. I.e.: insettings.pyadd:INSTALLED_APPS = [
  ...
  'django_scrubber',
  ...
]Scrubbing dataIn order to scrub data, i.e.: to replace DB data with anonymized versions,django-scrubbermust know which models and
fields it should act on, and how the data should be replaced.There are a few different ways to select which data should be scrubbed, namely: explicitly per model field; or globally
per name or field type.Adding scrubbers directly to model, matching scrubbers to fields by name:classMyModel(Model):somefield=CharField()classScrubbers:somefield=scrubbers.Hash('somefield')Adding scrubbers globally, either by field name or field type:# (in settings.py)SCRUBBER_GLOBAL_SCRUBBERS={'name':scrubbers.Hash,EmailField:scrubbers.Hash,}Model scrubbers override field-name scrubbers, which in turn override field-type scrubbers.To disable global scrubbing in some specific model, simply set the respective field scrubber toNone.Scrubbers defined for non-existing fields will raise a warning but not fail the scubbing process.Which mechanism will be used to scrub the selected data is determined by using one of the provided scrubbers
indjango_scrubber.scrubbers. See below for a list.
Alternatively, values may be anything that can be used as a value in aQuerySet.update()call (likeFuncinstances,
string literals, etc), or anycallablethat returns such an object when called with aFieldobject as argument.By default,django_scrubberwill affect all models from all registered apps. This may lead to issues with third-party
apps if the global scrubbers are too general. This can be avoided with theSCRUBBER_APPS_LISTsetting. Using this, you
might for instance split yourINSTALLED_APPSinto multipleSYSTEM_APPSandLOCAL_APPS, then
setSCRUBBER_APPS_LIST = LOCAL_APPS, to scrub only your own apps.Finally just run./manage.py scrub_datatodestructivelyscrub the registered fields.Arguments to the scrub_data command--modelScrub only a single model (format <app_label>.<model_name>)--keep-sessionsWill NOT truncate all (by definition critical) session data.--remove-fake-dataWill truncate the database table storing preprocessed data for the Faker library.Built-In scrubbersEmpty/NullThe simplest scrubbers: replace the field's content with the empty string orNULL, respectively.classScrubbers:somefield=scrubbers.Emptysomeother=scrubbers.NullThese scrubbers have no options.KeeperWhen running the validation or want to work in strict mode, you maybe want to actively decide to keep certain data
instead of scrubbing them. In this case, you can just definescrubbers.Keep.classScrubbers:non_critical_field=scrubbers.KeepThese scrubber doesn't have any options.HashSimple hashing of content:classScrubbers:somefield=scrubbers.Hash# will use the field itself as sourcesomeotherfield=scrubbers.Hash('somefield')# can optionally pass a different field name as hashing sourceCurrently, this uses the MD5 hash which is supported in a wide variety of DB engines. Additionally, since security is
not the main objective, a shorter hash length has a lower risk of being longer than whatever field it is supposed to
replace.LoremSimple scrubber meant to replaceTextFieldwith a static block of text. Has no options.classScrubbers:somefield=scrubbers.LoremConcatWrapper arounddjango.db.functions.Concatto enable simple concatenation of scrubbers. This is useful if you want to
ensure a fields uniqueness through composition of, for instance, theHashandFaker(see below) scrubbers.The following will generate random email addresses by hashing the user-part and usingfakerfor the domain part:classScrubbers:email=scrubbers.Concat(scrubbers.Hash('email'),models.Value('@'),scrubbers.Faker('domain_name'))FakerReplaces content with the help offaker.classScrubbers:first_name=scrubbers.Faker('first_name')last_name=scrubbers.Faker('last_name')past_date=scrubbers.Faker('past_date',start_date="-30d",tzinfo=None)The replacements are done on the database-level and should therefore be able to cope with large amounts of data with
reasonable performance.TheFakerscrubber requires at least one argument: the faker provider used to generate random data.
Allfaker providersare supported, and you can also register
your own custom providers.Any remaining arguments will be passed through to that provider. Please refer to the faker docs if a provider accepts
arguments and what to do with them.LocalesFaker will be initialized with the current djangoLANGUAGE_CODEand will populate the DB with localized data. If you
want localized scrubbing, simply set it to some other value.IdempotencyBy default, the faker instance used to populate the DB uses a fixed random seed, in order to ensure different scrubbings
of the same data generate the same output. This is particularly useful if the scrubbed data is imported as a dump by
developers, since changing data during troubleshooting would otherwise be confusing.This behaviour can be changed by settingSCRUBBER_RANDOM_SEED=None, which ensures every scrubbing will generate random
source data.LimitationsScrubbing unique fields may lead toIntegrityErrors, since there is no guarantee that the random content will not be
repeated. Playing with different settings forSCRUBBER_RANDOM_SEEDandSCRUBBER_ENTRIES_PER_PROVIDERmay alleviate
the problem.
Unfortunately, for performance reasons, the source data for scrubbing with faker is added to the database, and
arbitrarily increasingSCRUBBER_ENTRIES_PER_PROVIDERwill significantly slow down scrubbing (besides still not
guaranteeing uniqueness).When usingdjango < 2.1and working onsqlitea bug within django causes field-specific scrubbing (
e.g.date_object) to fail. Please consider using a different database backend or upgrade to the latest django version.Scrubbing third-party modelsSometimes you just don't have control over some code, but you still want to scrub the data of a given model.A good example is the Django user model. It contains sensitive data, and you would have to overwrite the whole model
just to add the scrubber metaclass.That's the way to go:Define your Scrubber classsomewherein your codebase (like ascrubbers.py)# scrubbers.pyclassUserScrubbers:scrubbers.Faker('de_DE')first_name=scrubbers.Faker('first_name')last_name=scrubbers.Faker('last_name')username=scrubbers.Faker('uuid4')password=scrubbers.Faker('sha1')last_login=scrubbers.Nullemail=scrubbers.Concat(first_name,models.Value('.'),last_name,models.Value('@'),models.Value(settings.SCRUBBER_DOMAIN))Set up a mapping between your third-party model and your scrubber class# settings.pySCRUBBER_MAPPING={"auth.User":"apps.account.scrubbers.UserScrubbers",}SettingsSCRUBBER_GLOBAL_SCRUBBERS:Dictionary of global scrubbers. Keys should be either field names as strings or field type classes. Values should be one
of the scrubbers provided indjango_scrubber.scrubbers.Example:SCRUBBER_GLOBAL_SCRUBBERS={'name':scrubbers.Hash,EmailField:scrubbers.Hash,}SCRUBBER_RANDOM_SEED:The seed used when generating random content by the Faker scrubber. Setting this toNonemeans each scrubbing will
generate different data.(default: 42)SCRUBBER_ENTRIES_PER_PROVIDER:Number of entries to use as source for Faker scrubber. Increasing this value will increase the randomness of generated
data, but decrease performance.(default: 1000)SCRUBBER_SKIP_UNMANAGED:Do not attempt to scrub models which are not managed by the ORM.(default: True)SCRUBBER_APPS_LIST:Only scrub models belonging to these specific django apps. If unset, will scrub all installed apps.(default: None)SCRUBBER_ADDITIONAL_FAKER_PROVIDERS:Add additional fake providers to be used by Faker. Must be noted as full dotted path to the provider class.(default: empty list)SCRUBBER_FAKER_LOCALE:Set an alternative locale for Faker used during the scrubbing process.(default: None, falls back to Django's default locale)SCRUBBER_MAPPING:Define a class and a mapper which does not have to live inside the given model. Useful, if you have no control over the
models code you'd like to scrub.SCRUBBER_STRICT_MODE:When strict mode is activated, you have to define a scrubbing policy for every field of every type defined inSCRUBBER_REQUIRED_FIELD_TYPES. If you have unscrubbed fields and this flag is active, you can't runpython manage.py scrub_data.SCRUBBER_REQUIRED_FIELD_TYPES:Defaults to all text-based Django model fields. Usually, privacy-relevant data is only stored in text-fields, numbers
and booleans (usually) can't contain sensitive personal data. These fields will be checked when runningpython manage.py scrub_validation.(default: (models.CharField, models.TextField, models.URLField, models.JSONField, models.GenericIPAddressField,
models.EmailField,))SCRUBBER_REQUIRED_FIELD_MODEL_WHITELIST:Whitelists a list of models which will not be checked duringscrub_validationand when
activating the strict mode. Defaults to the non-privacy-related Django base models.(default: ['auth.Group', 'auth.Permission', 'contenttypes.ContentType', 'sessions.Session', 'sites.Site',
'django_scrubber.FakeData',))SCRUBBER_MAPPING={"auth.User":"my_app.scrubbers.UserScrubbers",}(default: {})LoggingScrubber uses the default django logger. The logger name isdjango_scrubber.scrubbers.
So if you want to log - for example - to the console, you could set up the logger like this:LOGGING['loggers']['django_scrubber'] = {
    'handlers': ['console'],
    'propagate': True,
    'level': 'DEBUG',
}Making a new releasebumpversionis used to manage releases.Add your changes to theCHANGELOGand runbumpversion <major|minor|patch>, then push (including
tags)ChangelogAll notable changes to this project will be documented in this file.The format is based onKeep a Changelogand this project adheres toSemantic Versioning.[1.2.2] - 2023-11-04ChangedSetdefault_auto_fieldfordjango-scrubbertodjango.db.models.AutoFieldto prevent overrides from Django settings - Thanks @GitRon[1.2.1] - 2023-11-03Invalid[1.2.0] - 2023-04-01ChangedAdded scrubber validation - Thanks @GitRonAdded strict mode - Thanks @GitRon[1.1.0] - 2022-07-11ChangedInvalid fields on scrubbers will no longer raise exception but just trigger warnings nowAuthor list completed[1.0.0] - 2022-07-11ChangedMeta data for python package improved - Thanks @GitRon[0.9.0] - 2022-06-27AddedAdd functionality to scrub third party models like the Django user model, seehttps://github.com/RegioHelden/django-scrubber#scrubbing-third-party-models- Thanks @GitRonAdd tests for Python 3.10 - Thanks @costela[0.8.0] - 2022-05-01AddedAddkeep-sessionsargument to scrub_data command. Will NOT truncate all (by definition critical) session data. Thanks @GitRonAddremove-fake-dataargument to scrub_data command. Will truncate the database table storing preprocessed data for the Faker library. Thanks @GitRonAdd Django 3.2 and 4.0 to test matrixChangedRemove Python 3.6 from test matrixRemove Django 2.2 and 3.1 from test matrix[0.7.0] - 2022-02-24ChangedRemove upper boundary for Faker as they release non-breaking major upgrades way too often, please pin a working release in your own app[0.6.2] - 2022-02-08ChangedSupport faker 12.x[0.6.1] - 2022-01-25ChangedSupport faker 11.x[0.6.0] - 2021-10-18AddedAdd support to override Faker locale in scrubber settingsChangedPublish coverage only on main repository[0.5.6] - 2021-10-08ChangedPin psycopg2 in CI ti 2.8.6 as 2.9+ is incompatible with Django 2.2[0.5.5] - 2021-10-08ChangedSupport faker 9.x[0.5.4] - 2021-04-13ChangedSupport faker 8.x[0.5.3] - 2021-02-04ChangedSupport faker 6.x[0.5.2] - 2021-01-12ChangedAdd tests for Python 3.9Add tests for Django 3.1Support faker 5.xUpdate dev package requirements[0.5.1] - 2020-10-16ChangedFix travis setup[0.5.0] - 2020-10-16AddedSupport for django-model-utils 4.x.xChangedAdd compatibility for Faker 3.x.x, remove support for Faker < 0.8.0Remove support for Python 2.7 and 3.5Remove support for Django 1.x[0.4.4] - 2019-12-11Fixedadd the same version restrictions on faker to setup.py[0.4.3] - 2019-12-04Addedadd empty and null scrubbersChangedmakeLoremscrubber lazy, matching READMEFixedset more stringent version requirements (faker >= 3 breaks builds)[0.4.1] - 2019-11-16Fixedcorrectly clear fake data model to fix successive calls toscrub_data(thanksBenedikt Bauer)[0.4.0] - 2019-11-13AddedFakerscrubber now supports passing arbitrary arguments to faker providers and also non-text fields (thanksBenedikt BauerandRonny Vedrilla)[0.3.1] - 2018-09-10Fixed#9Hashscrubber choking on fields withmax_length=None- Thanks toCharlie Denton[0.3.0] - 2018-09-06AddedFinally added some basic tests (thanksMarco De Felice)Hashscrubber can now also be used on sqliteChangedBREAKING: scrubbers that are lazily initialized now receiveFieldinstances as parameters, instead of field
names. If you have custom scrubbers depending on the previous behavior, these should be updated. Accessing the
field's name from the object instance is trivial:field_instance.name. E.g.: if you havesome_field = MyCustomScrubberin any of your models'Scrubbers, this class must accept aFieldinstance as first parameter.
Note that explicitly intializing any of the built-in scrubbers with field names is still supported, so if you were
just using built-in scrubbers, you should not be affected by this change.related to the above,FuncFieldderived classes can now do connection-based setup by implementing theconnection_setupmethod. This is mostly useful for doing different things based on the DB vendor, and is used to
implementMD5()on sqlite (see added feature above)Ignore proxy models when scrubbing (thanksMarco De Felice)Expand tests to include python 3.7 and django 2.1[0.2.1] - 2018-08-14AddedOption to scrub only one model from the management commandSupport loading additional faker providers by config setting SCRUBBER_ADDITIONAL_FAKER_PROVIDERSChangedSwitched changelog format to the one proposed onKeep a Changelog[0.2.0] - 2018-08-13Addedscrubbers.Concat to make simple concatenation of scrubbers possible[0.1.4] - 2018-08-13ChangedMake our README look beautiful on PyPI[0.1.3] - 2018-08-13Fixed#1badly timed import - Thanks toCharlie Denton[0.1.2] - 2018-06-22ChangedUse bumpversion and travis to make new releasesrename project: django_scrubber → django-scrubber[0.1.0] - 2018-06-22AddedInitial release