Package: django-datawatch
Django DatawatchWith Django Datawatch you are able to implement arbitrary checks on data, review their status and even describe what to do to resolve them.
Think ofnagios/icingafor data.Check execution backendsSynchronousWill execute all tasks synchronously which is not recommended but the most simple way to get started.CeleryWill execute the tasks asynchronously using celery as a task broker and executor.
Requires celery 5.0.0 or later.Other backendsFeel free to implement other task execution backends and send a pull request.Install$pipinstalldjango-datawatchAdddjango_datawatchto yourINSTALLED_APPSCelery beat database schedulerIf the datawatch scheduler should be run using the celery beat database scheduler, you need to installdjango_celery_beat.Adddjango_datawatch.tasks.django_datawatch_schedulerto theCELERYBEAT_SCHEDULEof your app.
This task should be executed every minute e.g.crontab(minute='*/1'), see example app.Write a custom checkCreatechecks.pyinside your module.fromdatetimeimportdatetimefromcelery.schedulesimportcrontabfromdjango_datawatch.datawatchimportdatawatchfromdjango_datawatch.baseimportBaseCheck,CheckResponsefromdjango_datawatch.modelsimportResult@datawatch.registerclassCheckTime(BaseCheck):run_every=crontab(minute='*/5')# scheduler will execute this check every 5 minutesdefgenerate(self):yielddatetime.now()defcheck(self,payload):response=CheckResponse()ifpayload.hour<=7:response.set_status(Result.STATUS.ok)elifpayload.hour<=12:response.set_status(Result.STATUS.warning)else:response.set_status(Result.STATUS.critical)returnresponsedefget_identifier(self,payload):# payload will be our datetime object that we are getting from generate methodreturnpayloaddefget_payload(self,identifier):# as get_identifier returns the object we don't need to process it# we can return identifier directlyreturnidentifierdefuser_forced_refresh_hook(self,payload):payload.do_something().generateMust yield payloads to be checked. The check method will then be called for every payload..checkMust return an instance of CheckResponse..get_identifierMust return a unique identifier for the payload..user_forced_refresh_hookA function that gets executed when the refresh is requested by a user through theResultRefreshView.This is used in checks that are purely based on triggers, e.g. when a field changes the test gets executed.trigger check updatesCheck updates for individual payloads can also be triggered when related datasets are changed.
The map for update triggers is defined in the Check class' trigger_update attribute.trigger_update = dict(subproduct=models_customer.SubProduct)The key is a slug to define your trigger while the value is the model that issues the trigger when saved.
You must implement a resolver function for each entry with the name of get__payload which returns the payload or multiple payloads (as a list) to check (same datatype as .check would expect or .generate would yield).def get_subproduct_payload(self, instance):
    return instance.productExceptionsDatawatchCheckSkipExceptionraise this exception to skip current check. The result will not appear in the checks results.Run your checksA management command is provided to queue the execution of all checks based on their schedule.
Add a crontab to run this command every minute and it will check if there's something to do.$./manage.pydatawatch_run_checks
$./manage.pydatawatch_run_checks--slug=example.checks.UserHasEnoughBalanceRefresh your check resultsA management command is provided to forcefully refresh all existing results for a check.
This comes in handy if you changes the logic of your check and don't want to wait until the periodic execution or an update trigger.$./manage.pydatawatch_refresh_results
$./manage.pydatawatch_refresh_results--slug=example.checks.UserHasEnoughBalanceGet a list of registered checks$./manage.pydatawatch_list_checksClean up your databaseRemove the unnecessary check results and executions if you've removed the code for a check.$./manage.pydatawatch_clean_upSettingsDJANGO_DATAWATCH_BACKEND='django_datawatch.backends.synchronous'DJANGO_DATAWATCH_RUN_SIGNALS=TrueDJANGO_DATAWATCH_BACKENDYou can chose the backend to run the tasks. Supported are 'django_datawatch.backends.synchronous' and 'django_datawatch.backends.celery'.Default: 'django_datawatch.backends.synchronous'DJANGO_DATAWATCH_RUN_SIGNALSUse this setting to disable running post_save updates during unittests if required.Default: Truecelery task queueDatawatch supported setting a specific queue in release < 0.4.0With the switch to celery 4, you should use task routing to define the queue for your tasks, seehttp://docs.celeryproject.org/en/latest/userguide/routing.htmlMigrating from 3.x to 4.xIn4.x, the base check class supports two new methods (get_assigned_usersandget_assigned_groups). This is a breaking change as the oldget_assigned_userandget_assigned_groupmethods have been removed.This change allows checks to assign multiple users and groups to a check. If you have implemented custom checks, you need to update your code to return a list of users and groups instead of a single user and group. e.g.Fromfromdjango_datawatch.baseimportBaseCheckclassCustomCheck(BaseCheck):...defget_assigned_user(self,payload,result)->Optional[AbstractUser]:returnNone# or a userdefget_assigned_group(self,payload,result)->Optional[Group]:returnNone# or a groupTofromdjango_datawatch.baseimportBaseCheckclassCustomCheck(BaseCheck):...defget_assigned_users(self,payload,result)->Optional[List[AbstractUser]]:returnNone# or a list of usersdefget_assigned_groups(self,payload,result)->Optional[List[Group]]:returnNone# or a list of groupsCONTRIBUTEDev environmentDocker(19.03.0 or later)Compose plugin for DockerPlease make sure that no other container is using port 8000 as this is the one you're install gets exposed to:http://localhost:8000/SetupWe've included an example app to show how django_datawatch works.
Start by launching the included docker container.dockercomposeup-dThen setup the example app environment.dockercomposerun--rmappmigrate
dockercomposerun--rmapploaddataexampleThe installed superuser is "example" with password "datawatch".Run checksOpenhttp://localhost:8000/, log in and then go back tohttp://localhost:8000/.
You'll be prompted with an empty dashboard. That's because we didn't run any checks yet.
Let's enqueue an update.dockercomposerun--rmappdatawatch_run_checks--forceThe checks for the example app are run synchronously and should be updated immediately.
If you decide to switch to the celery backend, you should now start a celery worker to process the checks.dockercomposerun--rm--entrypointceleryapp-Aexampleworker-lDEBUGTo execute the celery beat scheduler which runs the datawatch scheduler every minute, just run:dockercomposerun--rm--entrypointceleryapp-Aexamplebeat--schedulerdjango_celery_beat.schedulers:DatabaseSchedulerYou will see some failed check now after you refreshed the dashboard view.Run the testsdockercomposerun--rmapptestRequirements upgradesCheck for upgradeable packages by runningdockercomposeup-d
dockercomposeexecapppip-checkTranslationsCollect and compile translations for all registered localesdockercomposerun--rmappmakemessages--no-location--all
dockercomposerun--rmappcompilemessagesMaking a new releasebumpversionis used to manage releases.Add your changes to theCHANGELOG, rundockercomposeexecappbumpversion<major|minor|patch>then push (including tags).