Package: django-mysql-compressed-fields
django-mysql-compressed-fieldsThis package providesCompressedTextField, a MySQL-specific
Django model field similar toTextFieldorCharFieldthat stores its
value in compressed form viazlib.In particular you can replace a TextField or CharField like:fromdjango.dbimportmodelsclassProjectTextFile(models.Model):content=models.TextField(blank=True)with:fromdjango.dbimportmodelsfrommysql_compressed_fieldsimportCompressedTextFieldclassProjectTextFile(models.Model):content=CompressedTextField(blank=True)such that the text value of the field is actually compressed in the database.String-based lookups are supported:html_files=ProjectTextFile.objects.filter(content__contains='<html')html_files=ProjectTextFile.objects.filter(content__startswith='<!DOCTYPE')html_files=ProjectTextFile.objects.filter(content__endswith='</html>')empty_html_files=ProjectTextFile.objects.filter(content__in=['','<html></html>'])Advanced manipulations with MySQL'sCOMPRESS(),UNCOMPRESS(), andUNCOMPRESSED_LENGTH()functions are also supported:fromdjango.db.modelsimportFfrommysql_compressed_fieldsimportUncompressedLengthfiles=ProjectTextFile.objects.only('id').annotate(content_length=UncompressedLength(F('content')))DependenciesDjango3.2 or later requiredMySQL5.7 or later required...and nothing else 🎉LicenseMITMigration StepsTo migrate an existing TextField or CharField to be a CompressedTextField:Install this package:pip3 install django-mysql-compressed-fieldsFind an existing Django model with an uncompressedTextFieldorCharFieldthat you want to compress. For example:fromdjango.dbimportmodelsclassProjectTextFile(models.Model):content=models.TextField(blank=True)Add a*_compressedsibling field that will be used to hold the compressed
version of the original field. Mark it asdefault=''. Include an explicitdb_column=...value:fromdjango.dbimportmodelsfrommysql_compressed_fieldsimportCompressedTextFieldclassProjectTextFile(models.Model):content=models.TextField(blank=True)content_compressed=CompressedTextField(blank=True,default='',# needed by Django when adding a fielddb_column='content_compressed',# pin column name)Generate a migration to add the compressed field:python3 manage.py makemigrationsGenerate a new empty migration in the same app where the field is defined,
which we will use to populate the compressed field:python3 manage.py makemigrations --empty __APP_NAME__Open the empty migration file. It should look something like:fromdjango.dbimportmigrationsclassMigration(migrations.Migration):dependencies=[('ide','0002_projecttextfile_content_compressed'),]operations=[]Edit theoperationsfield to use a RunPython step to populate
the compressed field from the uncompressed field:fromdjango.dbimportmigrationsfromdjango.db.modelsimportFfrommysql_compressed_fieldsimportCompressdef_populate_content_compressed(apps,schema_editor):ProjectTextFile=apps.get_model('ide','ProjectTextFile')# NOTE: Assumes "content" field is already UTF-8 encoded,#       because CompressedTextField assumes UTF-8 encoding.ProjectTextFile.objects.update(content_compressed=Compress(F('content')))classMigration(migrations.Migration):dependencies=[('ide','0002_projecttextfile_content_compressed'),]operations=[migrations.RunPython(code=_populate_content_compressed,reverse_code=migrations.RunPython.noop,atomic=False,)]Remove the original uncompressed field from the model,
leaving only the compressed field remaining:fromdjango.dbimportmodelsfrommysql_compressed_fieldsimportCompressedTextFieldclassProjectTextFile(models.Model):content_compressed=CompressedTextField(blank=True,default='',# needed by Django when adding a fielddb_column='content_compressed',# pin column name)Generate a migration to remove the uncompressed field:python3 manage.py makemigrationsRename the compressed field without the*_compressedsuffix
so that it now has the name of the original uncompressed field:fromdjango.dbimportmodelsfrommysql_compressed_fieldsimportCompressedTextFieldclassProjectTextFile(models.Model):content=CompressedTextField(blank=True,default='',# needed by Django when adding a fielddb_column='content_compressed',# pin column name)Generate a migration to rename the field:python3 manage.py makemigrationsWhen prompted whether the field was renamed, answery(for yes).You now have a compressed version of the original field. All done! 🎉SponsorThis project is brought to you byTechSmart, which seeks to inspire the
next generation of K-12 teachers and students to learn coding and create
amazing things with computers. We use Django heavily.API ReferenceAll classes and functions below should be imported directly frommysql_compressed_fields. For example:frommysql_compressed_fieldsimportCompressedTextFieldFieldsCompressedTextFieldclassCompressedTextField(encode_errors='strict', decode_errors='strict', **options)A large text field, stored compressed in the database.Generally behaves likeTextField. Stores values in the database using the
same database column type asBinaryField. The value is compressed in the
same format that MySQL'sCOMPRESS()function uses. Compression and
decompression is performed by Django and not the database.encode_errorscontrols how encoding errors are handled when saving the field.decode_errorscontrols how decoding errors are handled when loading the field. If'strict'(the default), a UnicodeError exception is raised. Other possible values are'ignore','replace', and any other name registered viacodecs.register_error(). SeeError Handlersfor details.If you specify amax_lengthattribute, it will be reflected in the
Textarea widget of the auto-generated form field. However it is not
enforced at the model or database level. Themax_lengthapplies to the
length of the uncompressed text rather than the compressed text.String-based lookups can be used with this field type.
Such lookups will transparently decompress the field on the database server.html_files=ProjectTextFile.objects.filter(content__contains='<html')html_files=ProjectTextFile.objects.filter(content__startswith='<!DOCTYPE')html_files=ProjectTextFile.objects.filter(content__endswith='</html>')empty_html_files=ProjectTextFile.objects.filter(content__in=['','<html></html>'])Note that F-expressions that reference this field type will always refer to
the compressed value rather than the uncompressed value. So you may need to
use the Compress() and Uncompress() database functions manually when working
with F-expressions.# Copy a TextField value (in utf8 collation) to a CompressedTextFieldProjectTextFile.objects.filter(...).update(content=Compress(F('name')))# Copy a CompressedTextField value to a TextField (in utf8 collation)ProjectTextFile.objects.filter(...).update(name=Uncompress(F('content')))# Copy a CompressedTextField value to a CompressedTextFieldProjectTextFile.objects.filter(...).update(content=F('content'))The default form widget for this field is adjango.contrib.admin.widgets.AdminTextareaWidget(a kind ofTextInput).Database functionsCompressThe MySQLCOMPRESS()function, usable inF() expressions.UncompressThe MySQLUNCOMPRESS()function, usable inF() expressions.UncompressedLengthThe MySQLUNCOMPRESSED_LENGTH()function, usable inF() expressions.compressdefcompress(uncompressed_bytes:bytes)->bytes:The MySQLCOMPRESS()function.uncompressdefuncompress(compressed_bytes:bytes)->bytes:The MySQLUNCOMPRESS()function.uncompressed_lengthdefuncompressed_length(compressed_bytes:bytes)->int:The MySQLUNCOMPRESSED_LENGTH()function.compressed_lengthdefcompressed_length(uncompressed_bytes:bytes,*,chunk_size:int=64*1000,stop_if_greater_than:Optional[int]=None)->int:Returns the length of COMPRESS(uncompressed_bytes).Ifstop_if_greater_thanis specified and a result greater thanstop_if_greater_thanis returned then the compressed length is
no less than the returned result.Running TestsInstallDocker.Install MySQL CLI tools:If macOS, install using brew:brew install mysql-client@8.0Otherwise install from source:https://downloads.mysql.com/archives/community/Add MySQL CLI tools to path:export PATH="/usr/local/opt/mysql-client@8.0/bin:$PATH"Start MySQL server:docker run --name ide_db_server -e MYSQL_DATABASE=ide_db -e MYSQL_ROOT_PASSWORD=root -p 127.0.0.1:8889:3306 -d mysql:8.0Run tests:cd tests/test_data/mysitepoetry installpoetry run python3 manage.py testRelease Notesv1.2.0Add theencode_errorsanddecode_errorsoptions toCompressedTextField.v1.1.0Fix to support Django 4.1.v1.0.1Add logo.v1.0.0Initial release.