Package: django-pgcrypto-fields
django-pgcrypto-fieldsdjango-pgcrypto-fieldsis aDjangoextension which relies uponpgcryptoto
encrypt and decrypt data for fields.Requirementspostgres withpgcryptoSupports Django 2.2.x, 3.0.x, 3.1.x and 3.2.xCompatible with Python 3 onlyLast version of this library that supportsDjango1.8.x, 1.9.x, 1.10.x
wasdjango-pgcrypto-fields2.2.0.Last version of this library that supportsDjango2.0.x and 2.1.x was
wasdjango-pgcrypto-fields2.5.2.InstallationInstall packagepipinstalldjango-pgcrypto-fieldsDjango settingsOur library support different crypto keys for multiple databases by
defining the keys in yourDATABASESsettings.Insettings.py:importosBASEDIR=os.path.dirname(os.path.dirname(__file__))PUBLIC_PGP_KEY_PATH=os.path.abspath(os.path.join(BASEDIR,'public.key'))PRIVATE_PGP_KEY_PATH=os.path.abspath(os.path.join(BASEDIR,'private.key'))# Used by PGPPublicKeyField used by default if not specified by the dbPUBLIC_PGP_KEY=open(PUBLIC_PGP_KEY_PATH).read()PRIVATE_PGP_KEY=open(PRIVATE_PGP_KEY_PATH).read()# Used by TextHMACField and PGPSymmetricKeyField if not specified by the dbPGCRYPTO_KEY='ultrasecret'DIFF_PUBLIC_PGP_KEY_PATH=os.path.abspath(os.path.join(BASEDIR,'tests/keys/public_diff.key'))DIFF_PRIVATE_PGP_KEY_PATH=os.path.abspath(os.path.join(BASEDIR,'tests/keys/private_diff.key'))# And add 'pgcrypto' to `INSTALLED_APPS` to create the extension for# pgcrypto (in a migration).INSTALLED_APPS=('pgcrypto',# Other installed apps)DATABASES={# This db will use the default keys above'default':{'ENGINE':'django.db.backends.postgresql_psycopg2','NAME':'pgcryto_fields','USER':'pgcryto_fields','PASSWORD':'xxxx','HOST':'psql.test.com','PORT':5432,'OPTIONS':{'sslmode':'require',}},'diff_keys':{'ENGINE':'django.db.backends.postgresql_psycopg2','NAME':'pgcryto_fields_diff','USER':'pgcryto_fields_diff','PASSWORD':'xxxx','HOST':'psqldiff.test.com','PORT':5432,'OPTIONS':{'sslmode':'require',},'PGCRYPTO_KEY':'djangorocks','PUBLIC_PGP_KEY':open(DIFF_PUBLIC_PGP_KEY_PATH,'r').read(),'PRIVATE_PGP_KEY':open(DIFF_PRIVATE_PGP_KEY_PATH,'r').read(),},}Generate GPG keys if using Public Key EncryptionThe public key is going to encrypt the message and the private key will be
needed to decrypt the content. The following commands have been taken from thepgcrypto documentation(see Generating PGP Keys with GnuPG).Generating a public and a private key (The preferred key type is "DSA and Elgamal".):$gpg--gen-key
$gpg--list-secret-keys

/home/bob/.gnupg/secring.gpg
---------------------------
sec2048R/212014-10-23
uidTestKey<example@example.com>
ssb2048R/422014-10-23


$gpg-a--export42>public.key
$gpg-a--export-secret-keys21>private.keyLimitationsThis library currently does not support Public Key Encryption private keys that are password protected yet. See Issue #89 to help implement it.Upgrading to 2.4.0 from previous versionsThe 2.4.0 version of this library received a large rewrite in order to support
auto-decryption when getting encrypted field data as well as the ability to filter
on encrypted fields without using the old PGPCrypto aggregate functions available
in previous versions.The following items in this library have been removed and therefore references in
your application to these items need to be removed as well:managers.PGPManageradmin.PGPAdminaggregates.*Fieldsdjango-pgcrypto-fieldshas 3 kinds of fields:Hash based fieldsPublic Key (PGP) fieldsSymmetric fieldsHash Based FieldsSupported hash based fields are:TextDigestFieldTextHMACFieldTextDigestFieldis hashed in the database using thedigestpgcrypto function
using thesha512algorithm.TextHMACFieldis hashed in the database using thehmacpgcrypto function
using a key and thesha512algorithm. This is similar to the digest version however
the hash can only be recalculated knowing the key. This prevents someone from altering
the data and also changing the hash to match.Public Key Encryption FieldsSupported PGP public key fields are:CharPGPPublicKeyFieldEmailPGPPublicKeyFieldTextPGPPublicKeyFieldDatePGPPublicKeyFieldDateTimePGPPublicKeyFieldTimePGPPublicKeyFieldIntegerPGPPublicKeyFieldBigIntegerPGPPublicKeyFieldDecimalPGPPublicKeyFieldFloatPGPPublicKeyFieldPublic key encryption creates a token generated with a public key to
encrypt the data and a private key to decrypt it.Public and private keys can be set in settings withPUBLIC_PGP_KEYandPRIVATE_PGP_KEY.Symmetric Key Encryption FieldsSupported PGP symmetric key fields are:CharPGPSymmetricKeyFieldEmailPGPSymmetricKeyFieldTextPGPSymmetricKeyFieldDatePGPSymmetricKeyFieldDateTimePGPSymmetricKeyFieldTimePGPSymmetricKeyFieldIntegerPGPSymmetricKeyFieldBigIntegerPGPSymerticKeyFieldDecimalPGPSymmetricKeyFieldFloatPGPSymmetricKeyFieldEncrypt and decrypt the data withsettings.PGCRYPTO_KEYwhich acts like a password.Django Model Field EquivalentsDjango FieldPublic Key FieldSymmetric Key FieldCharFieldCharPGPPublicKeyFieldCharPGPSymmetricKeyFieldEmailFieldEmailPGPPublicKeyFieldEmailPGPSymmetricKeyFieldTextFieldTextPGPPublicKeyFieldTextPGPSymmetricKeyFieldDateFieldDatePGPPublicKeyFieldDatePGPSymmetricKeyFieldDateTimeFieldDateTimePGPPublicKeyFieldDateTimePGPSymmetricKeyFieldTimeFieldTimePGPPublicKeyFieldTimePGPSymmetricKeyFieldIntegerFieldIntegerPGPPublicKeyFieldIntegerPGPSymmetricKeyFieldBigIntegerFieldBigIntegerPGPPublicKeyFieldBigIntegerPGPSymmetricKeyFieldDecimalFieldDecimalPGPPublicKeyFieldDecimalPGPSymmetricKeyFieldFloatFieldFloatPGPPublicKeyFieldFloatPGPSymmetricKeyFieldOther Django model fields are not currently supported. Pull requests are welcomed.UsageModel Definitionfromdjango.dbimportmodelsfrompgcryptoimportfieldsclassMyModel(models.Model):digest_field=fields.TextDigestField()digest_with_original_field=fields.TextDigestField(original='pgp_sym_field')hmac_field=fields.TextHMACField()hmac_with_original_field=fields.TextHMACField(original='pgp_sym_field')email_pgp_pub_field=fields.EmailPGPPublicKeyField()integer_pgp_pub_field=fields.IntegerPGPPublicKeyField()pgp_pub_field=fields.TextPGPPublicKeyField()date_pgp_pub_field=fields.DatePGPPublicKeyField()datetime_pgp_pub_field=fields.DateTimePGPPublicKeyField()time_pgp_pub_field=fields.TimePGPPublicKeyField()decimal_pgp_pub_field=fields.DecimalPGPPublicKeyField()float_pgp_pub_field=fields.FloatPGPPublicKeyField()email_pgp_sym_field=fields.EmailPGPSymmetricKeyField()integer_pgp_sym_field=fields.IntegerPGPSymmetricKeyField()pgp_sym_field=fields.TextPGPSymmetricKeyField()date_pgp_sym_field=fields.DatePGPSymmetricKeyField()datetime_pgp_sym_field=fields.DateTimePGPSymmetricKeyField()time_pgp_sym_field=fields.TimePGPSymmetricKeyField()decimal_pgp_sym_field=fields.DecimalPGPSymmetricKeyField()float_pgp_sym_field=fields.FloatPGPSymmetricKeyField()EncryptingData is automatically encrypted when inserted into the database.Example:>>> MyModel.objects.create(value='Value to be encrypted...')Hash fields can have hashes auto updated if you use theoriginalattribute. This
attribute allows you to indicate another field name to base the hash value on.fromdjango.dbimportmodelsfrompgcryptoimportfieldsclassUser(models.Model):first_name=fields.TextPGPSymmetricKeyField(max_length=20,verbose_name='First Name')first_name_hashed=fields.TextHMACField(original='first_name')In the above example, if you specify the optional original attribute it would
take the unencrypted value from the first_name model field as the input value
to create the hash. If you did not specify an original attribute, the field
would work as it does now and would remain backwards compatible.PGP fieldsWhen accessing the field name attribute on a model instance we are getting the
decrypted value.Example:>>> # When using a PGP public key based encryption
>>> my_model = MyModel.objects.get()
>>> my_model.value
'Value decrypted'Filtering encrypted values is now handled automatically as of 2.4.0. Andaggregatemethods are not longer supported and have been removed from the library.Also, auto-decryption is support forselect_related()models.fromdjango.dbimportmodelsfrompgcryptoimportfieldsclassEncryptedFKModel(models.Model):fk_pgp_sym_field=fields.TextPGPSymmetricKeyField(blank=True,null=True)classEncryptedModel(models.Model):pgp_sym_field=fields.TextPGPSymmetricKeyField(blank=True,null=True)fk_model=models.ForeignKey(EncryptedFKModel,blank=True,null=True,on_delete=models.CASCADE)Example:>>> import EncryptedModel
>>> my_model = EncryptedModel.objects.get().select_releated('fk_model')
>>> my_model.pgp_sym_field
'Value decrypted'
>>> my_model.fk_model.fk_pgp_sym_field
'Value decrypted'Hash fieldsTo filter hash based values we need to compare hashes. This is achieved by using
a__hash_oflookup.Example:>>> my_model = MyModel.objects.filter(digest_field__hash_of='value')
[<MyModel: MyModel object>]
>>> my_model = MyModel.objects.filter(hmac_field__hash_of='value')
[<MyModel: MyModel object>]Limitations.distinct('encrypted_field_name')Due to a missing feature in the Django ORM, usingdistinct()on an encrypted field
does not work for Django 2.0.x and lower.The normal distinct works on Django 2.1.x and higher:items=EncryptedFKModel.objects.filter(pgp_sym_field__startswith='P').only('id','pgp_sym_field','fk_model__fk_pgp_sym_field').distinct('pgp_sym_field')Workaround for Django 2.0.x and lower:fromdjango.dbimportmodelsitems=EncryptedFKModel.objects.filter(pgp_sym_field__startswith='P').annotate(_distinct=models.F('pgp_sym_field')).only('id','pgp_sym_field','fk_model__fk_pgp_sym_field').distinct('_distinct')This works because the annotated field is auto-decrypted by Django as aFfield and that
field is used in thedistinct().Migrating existing fields into PGCrypto FieldsMigrating existing fields into PGCrypto Fields is not performed by this library.  You will need to migrate the data
in a forwards migration or other means. The only migration that is supported except to create/activate the pgcrypto
extension in Postgres.Migrating data is complicated as there might be a few things to consider such as:the shape of the datavalidations/constrains done on the table/model/form and anywhere elseThe library has no way of doing all these guesses or to make all these decisions.If you need to migrate data from unencrypted fields to encrypted fields, three ways to solve it:When there's no data in the db it should be possible to start from scratch by recreating the dbWhen there's no data in the table it should be possible to recreate the tableWhen there's data or if the project is shared it should be possible to do it in a non destructive wayOption 1: No data is in the dbDrop the databaseSquash the migrationsRecreate the dbOption 2: No data in the tableCreate a migration to drop the tableCreate a new migration for the table with the encrypted fieldOptionally squash the migrationOption 3: Migrating in a non-destructive wayThe goal here is to be able to use to legacy field if something goes wrong.Part 1:Create new fieldWhen data is saved write both to legacy and new fieldCreate a data migration to cast data from legacy field to new fieldcheck existing data from legacy and new field are the same if possiblePart 2:Rename the fields and drop legacy fieldsUpdate the code to use only the new fieldSecurity LimitationsTaken direction from the PostgreSQL documentation:https://www.postgresql.org/docs/9.6/static/pgcrypto.html#AEN187024All pgcrypto functions run inside the database server. That means that all the
data and passwords move between pgcrypto and client applications in clear text. Thus you must:Connect locally or use SSL connections.Trust both system and database administrator.If you cannot, then better do crypto inside client application.The implementation does not resist side-channel attacks. For example, the time
required for a pgcrypto decryption function to complete varies among ciphertexts of
a given size.CHANGELOGMaster (unreleased)2.6.0Added support for Django 3.1.xUpdated requirements_dev.txtDropped support for Python 3.5Dropped support for Django below 2.2.x LTS releaseAdded support for BigIntegerFields (#169)Added documentation for migration existing data (#246)2.5.2Added support for Django 3.xUpdated requirements_dev.txt2.5.1Fixed regression in the definition of EmailPGPPublicKeyField (#77)Removed dead code (remove_validators and RemoveMaxLengthValidatorMixin)Updated requirements_dev.txt2.5.0Added new DecimalFields for both public and symmetric key (#64)Added new FloatFields for both public and symmetric key (#64)Added new TimeFields for both public and symmetric key (#64)Added support for different keys based on database (#67)2.4.0Added auto-decryption of all encrypted fields including FK tablesRemoved django-pgcrypto-fieldsaggregates,PGPManagerandPGPAdminas they are no longer neededAdded support forget_or_create()andupdate_or_create()(#27)Added support forget_by_natural_key()(#23)Added support foronly()anddefer()as they were not supported withPGPManagerAdded support fordistinct()(Django 2.1+ with workaround available for 2.0 and lower)Separated out dev requirements from setup.py requirementsUpdated packaging / setup.py to include long descriptionAdded AUTHORS and updated CONTRIBUTINGUpdated TravisCI to use Xenial to gain Python 3.7 in the matrix2.3.1Added__rangelookup for Date / DateTime fields (#59)Remove compatibility forDjango 1.8, 1.9, and 1.10(#62)Improvedsetup.py:check for Python 3.5+updated classifiersImprovedmakefile for release to usetwineAdded additional shields toREADMEUpdated Travis config to include Python 3.5 and 3.6Refactored lookups and mixins2.3.0Invalid release, bump to 2.3.12.2.0Merge.coveragercintosetup.cfgAdded.gitignorefileUpdated out-dated requirements (latest versions ofFlake8andpycodestyleare incompatible with each other)UpdatedREADMEwith better explanations of the fieldsImplemented DatePGPPublicKeyField and DateTimePGPPublicKeyField2.1.1Added support for Django 2.x+Updated requirements for testingUpdated travis config with Python 3.6 and additional environments2.1.0Thanks to @peterfarrell:Add support forDatePGPSymmetricKeyFieldandDateTimePGPSymmetricKeyFieldincluding support for serializing / deserializing django form fields.Add support for auto decryption of symmetric key and public key fields via
the PGPManager (and support for disabling it in the Django Admin via the PGPAdmin)2.0.0Remove compatibility forDjango 1.7.Add compatibility forDjango 1.10.AddDjango 1.9to the travis matrix.v1.0.1Exclude tests app from distributed package.v1.0.0Rename package frompgcrypto_fieldstopgcrypto.v0.7.0Makeget_placeholderaccepts a new argumentcompilerFix buggy import toAggregateNote: these changes have been done for django > 1.8.0.v0.6.4RemoveMaxLengthValidatorfrom email fields.v0.6.3Avoid settingmax_lengthon PGP fields.v0.6.2Allow/checkNULLvalues for:TextDigestField;TextHMACField;EmailPGPPublicKeyField;IntegerPGPPublicKeyField;TextPGPPublicKeyField;EmailPGPSymmetricKeyField.IntegerPGPSymmetricKeyField.TextPGPSymmetricKeyField.v0.6.1Fixcasting bug when sending negative values to integer fields.v0.6.0AddEmailPGPPublicKeyFieldandEmailPGPSymmetricKeyField.v0.5.0Rename the following fields:PGPPublicKeyFieldtoTextPGPPublicKeyField;PGPSymmetricKeyFieldtoTextPGPSymmetricKeyField;DigestFieldtoTextDigestField;HMACFieldtoTextHMACField.Add new integer fields:IntegerPGPPublicKeyField;IntegerPGPSymmetricKeyField.v0.4.0Make accessing decrypted value transparent. Fix bug when field had a string
representation ofmemoryviewfor PGP and keyed hash fields.v0.3.1FixEncryptedProxyFieldto select the correct item.v0.3.0AccessPGPPublicKeyFieldandPGPSymmetricKeySQLdecrypted values with
field's proxy_decrypted.Remove descriptor for field's name and raw value.v0.2.0Add hash based lookup forDigestFieldandHMACField.AddDigestField,HMACField,PGPPublicKeyAggregate,PGPSymmetricKeyAggregate.v0.1.0Add decryption through an aggregate class.Add encryption when inserting data to the database.