Package: django-data-sync
Django Data SyncEnables you to sync insensitive data (including FileField) between
environments with any Django backends (as long the model definitions are the
same) directly from admin interface.DISCLAIMERThere are no rigorous tests, yet. I haven't got the chance to explore how it
behaves with complex relationships.
So far, it has been used in two production grade
projects where the models are not too complex
(ManyToMany is not yet properly tested).Please use this at your own risk of data lost when syncing,
or you can do rigorous testing at your development phase.Featuresenables you to sync insensitive data between the same Django environments
(as long the model definitions are the same) directly from admin interfacerelation fields are supported (ManyToMany needs to be tested)synchronous sync or in background (only Cloud Tasks is supported)TO BE ADDEDadd support for ImageField and FileFieldDONEsupport multiple tasks queues, current plan is to support GCP Cloud TasksDONEadd authorization and authentication at data export endpointadd tests, since it's not possible to test with two Django servers locally
(or there is?), I have to think how to implement this correctlyMIGHT GET ADDEDcompare data in JSON for audit purposeadd support for another tasks queues so that is cloud platform agnosticInstallationpip install django-data-syncadddata_syncto yourINSTALLED_APPS......'data_sync',........Run migratepython manage.py migrate data_syncAdd to urlpatterns. Please do take note of the prefix URLs it will be used
later.
e.g. most likely we will include this inapiApp, thus the prefix is/api.path('',include('data_sync.urls')),PrefaceData Sync works by making use of natural key.
So I heavily recommend to readdjango docs on this topicbefore going further.You need to analyze your models and define their natural keys.
You can infer their natural keys usually from unique fields (and orunique_together).Fields that are defined as unique or inunique_togethercan be defined by
only using the field name  e.g. a Language is related to a Country.
In Language definition,
theunique_togetheris usually the Country + the Language's ISO 639-1.In code it'll look something like thisunique_together=(('country','code'),)Notice thatcountryin unique_together itself isabstract.
What definesa country?
In context ofunique_togetherit will be their ID, but ID is not natural key.
Country's natural key should be their ISO 2 code.So we can infer that natural key of Language, programmatically, is
the Country's ISO 2 code + the Language's ISO 639-1It'll look like this when you implement in codeclassLanguage(models.Model):defnatural_key(self):return(self.country.code,self.code,)In essence, natural key is usually combination of unique fields and orunique_together, but it needs to be moreverbose.UsageTo get Data Sync working, you need to register the models that want to be
synced.Only register insensitive models e.g. copy. Never sync sensitive
models e.g. User as it can expose very sensitive data.To register the models, you need to decorate them and use custom managers.fromdjango.dbimportmodelsimportdata_sync@data_sync.register_model(natural_key=['code'])classCountry(models.Model):objects=data_sync.managers.DataSyncEnhancedManager()code=models.CharField(max_length=2)# iso2........@data_sync.register_model(natural_key=['country.code','code'])classLanguage(models.Model):objects=data_sync.managers.DataSyncEnhancedManager()code=models.CharField(max_length=2)# iso 639-1........@data_sync.register_model(natural_key=['language.country.code','language.code','key'],fields=('value','key','language'),file_fields=('thumbnail',))classCopy(models.Model):objects=data_sync.managers.DataSyncEnhancedManager()language=models.ForeignKey(Language,on_delete=models.CASCADE)value=models.TextField()key=models.CharField(max_length=255)default=models.TextField()thumbnail=models.ImageField()........@data_sync.register_modelHere you need to define your natural key (read Preface for further topic).If natural key has value in related field, you need to use . (dot) notation.You can also pass argument tofieldsparameter if you want to limit which
fields that you want to be synced.To add FileField into Data Sync, add them intofile_fieldsparameter.DataSyncEnhancedManagerIt looks like manager initialization is done at class loading.
So adding custom manager programmatically might be considered hacky
(I would really like to love input on this).For now, I'm afraid you must define custom manager, with the default
attribute name i.e.objectsto use DataSyncEnhancedManager.DataSyncEnhancedManager just adds aget_by_natural_key methodand no other
else.Worker tasksWhen the code is deployed to GAE (and GAE only, flex and kube not supported yet),data_syncautomatically uses Cloud Tasks with the queue id ofdata_sync.Settings and ConfigurationData sync should work without additional settings
(if using synchronous mode which is the default).If you are deploying to GAE, it automatically uses Cloud Tasks,
which you should fill the optionals below.OptionalsDATA_SYNC_SERVICE_ACCOUNT_EMAILDefaults to `` (empty string). You need to fill this with GCP service account.
You can use GAE default service account.
It is needed for OIDC validation as recommended
by GCP.DATA_SYNC_FORCE_SYNCDefaults toFalse. Set this toTrueif you want to use synchronous
when deployed to GAE.DATA_SYNC_CLOUD_TASKS_QUEUE_IDDefaults todata_syncDATA_SYNC_CLOUD_TASKS_LOCATIONDefaults toeurope-west1DATA_SYNC_GOOGLE_CLOUD_PROJECTDefaults to value of env var ofGOOGLE_CLOUD_PROJECT.DATA_SYNC_GAE_VERSIONDefaults to value of env var ofGAE_VERSION, which is already set by GAE.DATA_SYNC_GAE_SERVICEDefaults to value of env var ofGAE_SERVICE, which is already set by GAE.Data SourceData Source holds information about an environment from which you want your
data to be synced.The URL is dependant on where and how you include thedata_sync.urlsat
installation phase.For example, if you includedata_sync.urlsin yourapiApp urlpatterns,
then the URL in data source must be appended with yourapiURL.
Thus it might look something like thishttps://example.com/api.If you includedata_sync.urlsin your rooturls, then Data Source URL will
look like thishttps://example.com.Do not include endslash.The SyncTo do a sync, simply create a Data PullCompatibilityPython 3.7, Django 2.2 and upTestingNo automated tests (yet.....).To test locally, you can spawn two django servers with different ports and
different database and set the Data Source accordingly.