Package: django-joblog
django-joblog v0.2.2A generic django-utility that helps to log stuff to the database.InstallationUsageParallelismContextDummyJobLoggerUsing the ModelConfigurationDB aliasLive updatesPing modeTestingThis repoOverviewfromdjango_joblogimportJobLoggerwithJobLogger("task-name")aslog:log.log("task started")if1!=2:log.error("The impossible happened!")The following information is stored to the database for further inspection:the task's namethe count of invocation for the specific taskstart-timeend-timedurationany log or error outputthe exception trace, for exception occurring inside thewith-blockThis can be useful in conjuction with cronjobs and asynchronous tasks with, e.g., these libraries:django-kronos,django-rq,rabbitmq...Installationpipinstalldjango-joblogThen adddjango_joblogtoINSTALLED_APPSin your djangosettings.pyand callmanage.py migrate.RequirementsPython2 or 3Django>= 1.10UsageParallelismBy default, jobs are not allowed to run in parallel. This can be changed withparallel=Truein
theJobLoggerconstructor.If you start a JobLogger while a job with the same name is already
running, adjango_joblog.JobIsAlreadyRunningErroris raised.Additionally, a job log entry
in the database will be created withblockedstate.For example, you might have a cronjob that runs every minute and looks for open tasks in the
database. If you wrap the task in aJobLoggeryou can be sure, that the tasks are not
worked on in parallel:fromdjango_joblogimportJobLogger,JobIsAlreadyRunningErrordefcronjob_open_task_worker():ifopen_tasks():withJobLogger("work-open-tasks")aslog:work_open_tasks(log)# to avoid the error message on multiple invocation:defcronjob_open_task_worker():ifopen_tasks():try:withJobLogger("work-open-tasks")aslog:work_open_tasks(log)exceptJobIsAlreadyRunningError:passContextTo change the logging-context within a job, useJobLoggerContext.
It might help to spot at which point an output is generated or an exception is thrown.fromdjango_joblogimportJobLogger,JobLoggerContextwithJobLogger("pull-the-api")aslog:credentials=get_credentials()log.log("using user%s"%credentials.name)withJobLoggerContext(log,"api"):api=Api(credentials)log.log("connected")withJobLoggerContext(log,"submit"):api.submit(data)log.log("%sitems submitted"%len(data))withJobLoggerContext(log,"check result"):log.log(api.check_result())The log output in database will look like this:using user Herbert
api: connected
api:submit: 42 items submitted
api:check result: 23 items updatedAn exception caught by the error log might look like this:api:submit: IOError - Status code 404 returned for url https://my.api.com/submit
 File "/home/user/python/awesome_project/api/Api.py, line 178, in Api._make_request
   self.session.post(url, data=params)
 File "/home/user/python/awesome_project/api/Api.py, line 66, in Api.submit
   self._make_request(url, params)
 File "/home/user/python/awesome_project/main.py, line 12
   api.submit(data)As can be seen, aJobLoggerContextdoes not pop it's name from the context stack
in case of an exception! Which means, catching exceptions within higher context levels
than where those exceptions where raised does not leave a valid context stack if
you resume work after the caught exception.inspecting exceptionsAn exception is kept in theexceptionproperty of theJobLoggerinstance:withJobLogger("test-exception")asjob:raiseValueError("This was bad")assertisinstance(job.exception,ValueError)assert"This was bad"==str(job.exception)assertjob.tracebackDummyJobLoggerYou can use theDummyJobLoggerclass to provide logging without storing stuff to the database.
This might be useful for debugging purposes, or if you run a function as amanage.py-task but
need database logging only for cronjobs.In general, functions can be designed to work with aJobLoggerbut do notrequireit.fromdjango_joblogimportJobLogger,DummyJobLoggerdefbuy_eggs(log=None):log=logorDummyJobLogger()log.log("Gonna buy some eggs!")...defcronjob_invokation():withJobLogger("buy-eggs")aslog:buy_eggs(log)defdebug_invokation():buy_eggs()Using the modelBy default, there is a django admin view for theJobLogModel.
You can find the model, as usual, indjango_joblog.models.
Please check the filedjango_joblog/models.pyfor the specific fields. It's nothing special.ConfigurationThedjango_joblogapp can be configured with an object in your django projectsettings.py,
for example:JOBLOG_CONFIG={# name of alternate database connection, to circumvent transactions on default connection"db_alias":"joblog",# enable .log and .error to write to database immediately"live_updates":True,# enable a constant update of job state - to check for jobs which went away without notice"ping":True,"ping_interval":1,# always print to console during jobs"print_to_console":True}The whole object and all of it's fields are optional.db_aliasdb_aliasdefines an alternative name for the database connection.
This name must be present in theDATABASEsetting.One does not normally need to define this setting unless you want to make sure thatLive updatesor using thePing modealways work even when transactions
are used inside the job body. Consider this example:fromdjango.dbimporttransactionfromdjango_joblogimportJobLoggerwithJobLogger("my-job")asjob:job.log("Outside transaction")withtransaction.atomic():job.log("Inside transaction")# ...other stuff...If you are usingLive updatesand need to make sure that the second log
("Inside transaction") is immediately stored to the database you need to define a second
database connection. It can just be a copy of the'default'database setting.live updatesSettinglive_updatestoTruewill store the current log and error texts as long with the current
job duration to database wheneverJobLogger.log()orJobLogger.error()are called.pingSettingpingtoTruewill spawn a separate thread when callingwith JobLogger(...)that will
constantly update the joblog database with the current log text, error text and duration.
The update-interval is configured withping_intervalin seconds.Normally, if a job exits unexpectedly (segfault, power-off, restart of vm, etc..) it's state in
the database will stayrunningforever.
New jobs with the same name will be blocked from execution.However, enablingpingmode will make sure, that if a job (in database) who'sdurationis
yet undefined or larger than theping_intervalcan be considered stopped.To set those dangling job's database state tovanisheduse:./manage.pyjoblog_cleanuporfromdjango_joblog.modelsimportJobLogModelJobLogModel.cleanup()TestingUnit-tests areDjango-styleand are placed indjango_joblog/tests/.Note that theparalleltests will fail with theSqlitebackend, because of database-locking.The repositoryThe repocontains a whole django project (django_joblog_project)
for ease of development.setup.pyonly exports thedjango_joblogapp.The default database backend is configured toMySQL.To start the runserver or run the tests within the repo, open mysql console:CREATEUSER'django_logs_user'@'localhost'IDENTIFIEDBY'django_logs_pwd';CREATEDATABASEdjango_logs_testCHARACTERSETutf8COLLATEutf8_general_ci;GRANTALLONdjango_logs_test.*TO'django_logs_user'@'localhost';GRANTALLONdjango_logs_test_test.*TO'django_logs_user'@'localhost';Then alternatively, depending on the python version:pipinstallMySQL-python# for python 2pipinstallPyMySQL# for python 3pipinstallmysqlclient#   or alternativelyAnd finally:./manage.pytest# or./manage.pymigrate
./manage.pyrunserverChangelogv0.2.5 - Nov/2019addJobLogger.exceptionproperty for inspection after the runv0.2.4 - Apr/2019add 'blocked' state entry on failed parallel job runsv0.2.3fix joblog model accessv0.2.2addJOBLOG_CONFIGsettingaddlive_updatesandpingmodev0.1.1 - Aug/2018add search and filter to admin viewv0.1.0 - Jul/2018make printing to console optionaladd django unit-testsv0.0.1Copy-pasted together from various private projects, 'sanitized' and repackaged for PyPi