Package: django-forcedfields
SummaryA Python module that provides a set of custom, specialized Django model fields.While I haved worked with Django’s ORM for some time and have enjoyed many of its features for
simple use cases, I find myself increasingly impeded, annoyed, and dissatisfied by its limitations
in complex applications. One glaring problem in my eyes is the ORM’s lack of semantic database field
data types and modifiers.For example, an eight-character varchar field that can be null and that has a default value of
“elegy” willnotresult in the MySQLDDL:VARCHAR(8) DEFAULT 'elegy' NULLbut simply as:VARCHAR(8) NULLWhile this varchar example may not be the most egregious, it nonetheless illustrates the almost
complete reliance upon the application and its ORM for behavior that should be handled, and indeed
is best handled, by the database management system itself.Databases should be as self-documenting and semantic as possible, independent of any application
code, ORM models, or documentation. I will not compromise this principle for the sake of an ORM’s
conveniences. To this end, I have begun to create these custom Django model fields to force Django
to issue the most specific and complete DDL statements possible. It is my goal with these and future
fields to shift responsibility from the application ORM to the underlying database wherever possible
while maintaining a consistent and complete ORM interface and database backend abstraction.Installationpip install [--user] django-forcedfieldsExample Usageimport django_forcedfields as forcedfieldsor:from django_forcedfields import TimestampFieldFieldsFixedCharFieldclass FixedCharField(max_length=None, **options)This field extends Django’sCharField.This field inherits all functionality and interfaces from Django’s standard CharField but, rather
than producing aVARCHARfield in the database, the FixedCharField creates aCHARfield. The
parent CharField class’ keyword argumentmax_lengthis retained and, when passed, specifies theCHARfield’s max length just like it does for theVARCHARimplementation. TheCHARdata
type is supported on all RDBMS in common use with Django.In addition, if a FixedCharField on a model is not given an explicit value and no default field
value has been explicitly defined, aNULLvalue will be inserted on Model.save(). This is in
contrast to Django’s standard CharField which incorrectly attempts to insert an empty string in such
a case. Ideally, with no explicit value and no default value, an integrity error would be raised by
the database but Django’s ORM absolutely requires a value for all fields inINSERToperations.
It is impossible to simply omit a database column’s value in anINSERTstatement.A note here on Django’sadmonition on null values with text fields: Django is wrong.NULLmeans
unknown data, an empty string means an empty string. Their meanings aresemantically differentby
definition. Setnull=Trueon text fields when your use case warrants it. That is, when you may
have a complete absence of data as well as the need to record an empty string. Google this topic
for more analysis.TimestampFieldclass TimestampField(auto_now=False, auto_now_add=False, auto_now_update=False, **options)This field extends Django’sDateTimeField.This field supports allDateTimeField keyword argumentsand adds a newauto_now_updateargument.TimestampField.auto_now_updateauto_now_updateis a boolean that, when True, sets a new timestamp field value on update
operationsonly, not on insert.This option is mutually exclusive withauto_now.Warning:When using the MySQL backend, the databaseTIMESTAMPfield will also be updated
whenauto_noworauto_now_updateis enabled and when calling QuerySet.update(). In
constrast, Django’s DateField and DateTimeField only set current timestamp underauto_nowwhen
calling Model.save().
This modified behavior is due to the declaration ofON UPDATE CURRENT_TIMESTAMPin the
TimestampField’s column definition DDL.Like its parent DateTimeField, the TimestampField’s optionsauto_now,auto_now_add, andauto_now_updatewill forcibly overwrite any manually-set model field attribute values when
enabled and when their conditions are triggered. The value will be the Django ORM database functionNow()rather than a datetime
instance since the value will have been generated by the database server and must therefore be
retrieved with a separate query.Naturally, when designing a system field instead of a user data field, the need to offload
responsibility to the underlying database becomes greater. If the data is for system and metadata
purposes, then it increases consistency and data integrity to delegate field value management to the
system itself.A timestamp is well-suited to record system and database record metadata such as record insert and
update times. Due to the database data type features, it is also ideal when storing a fixed point in
time, independent of time zone. Although the creation of the TimestampField was largely motivated by
the need for an ORM abstraction for metadata fields, it can also be used just like its parent
DateTimeField as long as one understands the data type’s different advantages and limitations.Instead of DateTimeField’s reliance onDATETIMEand similar data types, the TimestampField usesTIMESTAMPdata type and other data types that do not store time zone information. The data type
changes can be seen in the following table:databaseDateTimeField data typeTimestampField data typeMySQLDATETIMETIMESTAMPPostgreSQLTIMESTAMP WITH TIMEZONETIMESTAMP WITHOUT TIME ZONESQLiteDATETIMEDATETIMEAlso note that standard DDL modifiers such asDEFAULT CURRENT TIMESTAMPand non-standard ones
such as MySQL’sON UPDATE CURRENT_TIMESTAMPare used when the corresponding options on a
TimestampField instance are enabled.Database Engine ConsiderationsWhen using TimestampField, one must be aware of certain database engine behavior defaults and
configurations. An ORM is usually designed to abstract, as much as is practical and prudent, the
differences between the underlying databases. In this case, however, the abstraction leaks. Consider
the following timestamp column DDL:TIMESTAMP NOT NULLNote the lack of aDEFAULTclause. One would expect, upon attempting to insert aNULLvalue
or failing to provide a value for the column altogether, that some sort of constraint or integrity
exception would be raised. Indeed, this behavior adheres to the principle of least astonishment and
is the standard behavior of both SQLite and PostgreSQL. BothSQLiteandPostgreSQLimplicitly assignDEFAULT NULLto column definitions with no explicitDEFAULTclause.MySQLMySQL requires a specific configuration to achieve the same standard behavior. The following
configuration options affectTIMESTAMPcolumns:strict modeNO_ZERO_DATENO_ZERO_IN_DATEexplicit_defaults_for_timestampAt minimum, MySQL requires that both strict mode andexplicit_defaults_for_timestampare
enabled forTIMESTAMPbehavior to conform to standards. If one attempts to omit a value for theTIMESTAMP NOT NULLcolumn, a “ERROR 1364 (HY000): Field <field_name> doesn’t have a default
value” is emitted and if one attempts to insert aNULLvalue, a “ERROR 1048 (23000): Column
<field_name> cannot be null” is emitted. As of version MySQL 5.7, strict mode is enabled by default
butexplicit_defaults_for_timestampis not.MariaDBMariaDB, on the other hand, applies the same configuration parameters in a different way and its
logic as it relates toTIMESTAMP NOT NULLis less clear and, dare I say, erroneous. Assuming
identical configuration (strict mode andexplicit_defaults_for_timestampenabled), MariaDB
raises “ERROR 1364 (HY000): Field <field_name> doesn’t have a default value” on insert value
omission but successfully accepts aNULLvalue with no error and stores the results ofCURRENT_TIMESTAMP()in the field instead.In an attempt to bring MariaDB in line with the standard, I also testedNO_ZERO_DATEandNO_ZERO_IN_DATE. As long as bothexplicit_defaults_for_timestampandNO_ZERO_DATEorNO_ZERO_IN_DATEare enabled, it is impossible to create a table containing theTIMESTAMP NOT NULLcolumn as theCREATE TABLEstatement fails with “ERROR 1067 (42000):
Invalid default value for <field_name>”. This suggests that not only is theDEFAULTvalue
validated during DDL statements, but MariaDB is also attempting to implicitly define a zero valueDEFAULTvalue on theTIMESTAMPfield as the same error is raised whenDEFAULT'0000-00-0000:00:00'is explicitly defined. This is nonstandard, erroneous behavior and
conflicts with that of MySQL. From theMySQL documentation:TIMESTAMPcolumns explicitly declared with theNOT NULLattribute and without an
explicitDEFAULTattribute are treated as having no default value.From the same documentation page, the following governsINSERToperations under these
conditions:For inserted rows that specify no explicit value for such a column, the result depends on the
SQL mode. If strict SQL mode is enabled, an error occurs. If strict SQL mode is not enabled, the
column is declared with the implicit default of ‘0000-00-00 00:00:00’ and a warning occurs. This
is similar to how MySQL treats other temporal types such as DATETIME.The DDL validation failure may have something to do with theseINSERTrules.It is impossible for MariaDB’sTIMESTAMPfields to behave in a standard way when dealing withTIMESTAMP NOT NULLcolumns. I foundthis bug reportfor MariaDB but it appears that the work has ceased
and the fix has not been merged into the target release. All tests were performed on MariaDB 10.2
and 10.3.ConclusionI now have a choice to make: do I cause TimestampField to step aside and let the user more directly
experience the effects of the underlying database engine’s configuration or do I attempt to abstract
the behavior differences as much as possible? Given the spirit and goal of this library, I have
opted for less abstraction and have removed any additional, artificial normalization of database
engine behavior in these field classes. I am certainly open to discussion on this point so please
don’t hesitate to open communication with me or point out any errors in my testing.Given MariaDB’s deviation from standards, this package’s unit tests are performed using MySQL and
testing on MariaDB is disabled until further notice.As an aside, please note that many inconsistent behaviors between database engines can be mitigated
or even eliminated by explicitly defining field keyword arguments such asdefault,null,
etc., causing more explicit DDL SQL to be generated by Django in the resulting migrations and SQL.DevelopmentTo set up the development environment, a Vagrantfile is included. InstallVagrantand:vagrant upOnce Vagrant has completed provisioning,vagrant sshinto the box and start the database servers
against which to run the test suite:docker-compose up -dFinally, run the tests with:make testsThe Vagrant machine is provisioned to use the UTC time zone to facilitate tests. If you elect to run
tests outside of the Vagrant machine, be aware that certain tests assume identical time, date, and
time zone settings between all database engines. SQLite defaults to the host’s localtime while the
Docker containers use the host’s clock and default to the UTC time zone.In this project, I use PEP8 andGoogle’s Python style guide. Pylint doesn’t play nicely with some of the
styles. A few notes on pylint:bad-continuationIgnore most of these. Google style guide allows for a 4-space hanging indent with nothing on
first line.Example:indentationOracle SupportThe FixedCharField should work on Oracle but the TimestampField will default to DateTimeField
database field data types when used with Oracle. I neither implemented functionality for nor tested
on Oracle for a few reasons:It is too difficult to get an Oracle server instance against which to test. As one can see, I use
lightweight Docker containerized services to run the test databases. To use Oracle, one needs to
provide the Oracle installation binaries. To get the binaries, one needs to sign in to Oracle’s
web site for the privilege of downloading over 2.5 gigabytes. Too much unnecessary pain, not
enough return. If you use Oracle products, I sympathize and may god have mercy on your soul.https://github.com/oracle/docker-images/tree/master/OracleDatabaseOracle seems to berarely used with Django.I hate Oracle products and Oracle as an entity.Changelogv1.0Automatic values fromauto_now,auto_now_add, andauto_now_updateare no longer
generated in the application usingdatetime.datetime.now()ordjango.utils.timezone.now().CURRENT_TIMESTAMPgeneration is now performed by the database using the Django database
functiondjango.db.models.functions.Now.All fields now cause the ORM to issue explicitDEFAULTclauses in column DDL statements where
previously the ORM always omittedDEFAULTclauses from column definitions.DEFAULTclauses
will be defined in DDL if Field.has_default() returns True. This behavior naturally includes the
generation ofDEFAULT NULLin the column DDL if the field’sdefaultoption is set
toNone.If no kwargs (options) are passed to TimestampField, noDEFAULTclause is generated in the
column DDL for MySQL. Previously, aDEFAULT NULLorDEFAULT 0clause was output in the DDL
to disable MySQL’s defaultTIMESTAMPbehavior. Howver, defaultTIMESTAMPbehavior varies
according to certain server system variables and, depending upon configuration, it may be
completely valid to omit aDEFAULTclause altogether.FixedCharField will now attempt to insertNULLif no value is defined on the model’s field
attribute and no explicit field default value has been defined. This behavior is in contrast to
Django’s standard CharField which always attempts to (incorrectly) store an empty string in such a
case.