Package: rubydj-pyworker
rubydj-pyworkerA pure Python 3.x worker for Ruby-based DelayedJobs.
Note that Python 2.7 is not supported.Why rubydj-pyworker?If you have a scientifcRuby on Railsapplication,
you may find yourself requiring background jobs to run in Python
to leverage all the scientific libraries that Python is famous for.
So instead of creating wrapper Ruby jobs that dosystemcalls to Python,
why not implement the whole job in Python and launch a Python worker
to dispatch only those jobs?
Also, with the microservice architecture and containers world,
it is not practical to deploy a fat container with both Ruby and Python
installed. It is much more cleaner to deploy 2 separate containers.pyworkers offers you theWorkerclass that does all the background job
dispatching, scheduling and error handling, plus aJobclass that you
should subclass to implement your pure python jobs. It basically replicates
thedelayed_jobRuby gem behavior.InstallationFrom Pypi:pip install rubydj-pyworkerYou can also pick up any version fromPypithat supports Python 3.x (>= 1.0.0).From Github branch:pip install git+https://github.com/rayyansys/pyworker.git@<branch_name>#egg=rubydj-pyworkerUsageThe simplest usage is creating a worker and running it:frompyworker.workerimportWorkerdbstring='postgres://user:password@host:port/database'w=Worker(dbstring)w.run()This will create a worker that continuously polls the specified database
and pick up jobs to run. Of course, no jobs will be recognized so one should
declare jobs before calling the worker.Example job:frompyworker.jobimportJobclassMyJob(Job):def__init__(self,*args,**kwargs):super(MyJob,self).__init__(*args,**kwargs)defrun(self):self.logger.info("Running MyJob.run")# optionaldefbefore(self):self.logger.debug("Running MyJob.before hook")# optionaldefafter(self):self.logger.debug("Running MyJob.after hook")# optionaldefsuccess(self):self.logger.debug("Running MyJob.success hook")# optionaldeferror(self):self.logger.debug("Running MyJob.error hook")# optionaldeffailure(self):self.logger.debug("Running MyJob.failure hook")Once this example job is declared, the worker can recognize it and
will call itsrunmethod once it is picked up, optionally with the
specified hooks.ConfigurationBefore calling therunmethod on the worker, you have these
configuration options:# seconds between database polls (default 10)w.sleep_delay=3# maximum attempts before marking the job as permanently failing (default 3)w.max_attempts=5# maximum run time allowed for the job, before it expires (default 3600)w.max_run_time=14400# queue names to poll from the datbase, comma separated (default: 'default')w.queue_names='queue1,queue2'You can also provide a logger class (fromloggingmodule) to have full control on logging configuration:importlogginglogging.basicConfig()logger=logging.getLogger('pyworker')logger.setLevel(logging.INFO)w=Worker(dbstring,logger)w.run()MonitoringWorkers can be monitored usingNewrelic. All you need
to do is to create a free account there, then add the following to your
environment variables:NEW_RELIC_LICENSE_KEY=<your_newrelic_license_key>NEW_RELIC_APP_NAME=<your_newrelic_app_name>All jobs will be reported under theBackgroundTaskcategory. This includes
standard metrics like throughput, response time (job duration), error rate, etc.
Additional transaction custom attributes are also reported out of the box:jobName: the name of the job classjobId: the id of the delayed job in the databasejobQueue: the queue name of the jobjobAttempts: the number of attempts for the jobjobLatency: the time between the job creation and the time it was picked up by the workerIf you wish to automatically report additional attributes from the delayed job table
(e.g. the priority of the job or any other custom fields), you can do so by
providing a list of fields to the worker:worker=Worker(dbstring,logger=logger,extra_delayed_job_fields=['priority','custom_field'])Columns of typesstring/text,int,floatandboolare reported as is.json/jsonbtypes are expanded into separate attributes. All other fields are converted
to JSON strings.You can also automatically prefix all attributes with a string before reporting:worker=Worker(dbstring,logger=logger,reported_attributes_prefix='myapp.')If you wish to report additional custom attributes from your job, you can do so
by calling thereporterobject that is available in the job instance:classMyJob(Job):def__init__(self,*args,**kwargs):super(MyJob,self).__init__(*args,**kwargs)defrun(self):...self.reporter.report(my_custom_attribute='my_custom_value',another_custom_attribute='another_custom_value',...)...The prefix will be applied to all attributes reported from the job as well
as the camel case conversion. If you wish to skip both, you can use thereport_rawfunction instead ofreport.In all cases, two additional attributes are reported:error: a Boolean indicating whether the job has failed or notjobFailure: a Boolean indicating whether the job has permanently failed or notTheerrorattribute is reported as is, i.e. without prefix or camel case conversion.LimitationsOnly supports Postgres databasesNo read ahead option, picks up one job at a timeAssumes UTC timezone in the databaseNo access to your Ruby classes, you should implement all your logic from scratch in PythonReads only raw attributes of jobs from the database (job table columns), no relationsAssumes you only need to call therunmethod in your job with no argumentsContributeInstall the code for development:git clone https://github.com/rayyansys/pyworker.git
cd pyworker
python setup.py developDo your changes, then send a pull request.Testpip install -r requirements-test.txt
pytestPublishIncrement the version number insetup.pyInstalltwineandwheel:pip install twine wheel. You may need to upgrade pip first:pip install --upgrade pipCreate the distribution files:python setup.py sdist bdist_wheelOptionally upload toTest PyPias a dry run:twine upload -r testpypi dist/*. You will need a separate account thereUpload toPyPi:twine upload dist/*Enter your PyPi username and password when prompted.LicenseReleased under the MIT License.