Saravanan666/pypi-org huggingface dataset:
Package: celery-dispatcher
celery-dispatcherAn extension for celery to dispatch large amount of subtasks within a main task, and process the results separately.Installationpip install celery-dispatcherUsageNOTICE:celery-dispatcheruse tls to store running info, so the pool implementation using coroutines(like eventlet/gevent) can not be usedFirstly, yield subtask and its parameters in tuple from the main task by following order:task: signatureargs: tupple/listkwargs: dictoptions: dictThen register the result handler for each subtask using signal:from celery import shared_task
from celery_dispatcher import dispatch
from celery_dispatcher.signals import subtask_success

@shared_task
def sqrt(i):
    return i * i

@dispatch
@shared_task
def calc():
    for i in range(10):
        yield sqrt, (i,)

@subtask_success.connect(sender=calc)
def handle_result(root_id, task_id, retval, **kwargs):
    print(retval)Or register in the decorator directly:from celery import shared_task
from celery_dispatcher import dispatch

@shared_task
def sqrt(i):
    return i * i

def handle_result(root_id, task_id, retval, **kwargs):
    print(retval)

@dispatch(receiver=handle_result)
@shared_task
def calc():
    for i in range(10):
        yield sqrt, (i,)If you want to revoke all subtasks, call revoke with specific signal:import signal
from celery import shared_task
from celery_dispatcher import dispatch

@shared_task
def subtask(i):
    ...

@dispatch
@shared_task
def maintask():
    for i in range(10):
        yield subtask, (i,)

result = maintask.delay()
result.revoke(terminate=True, signal=signal.SIGUSR1)OptionsThedispatchaccepts the following parameters:options: dictreceiver: callablebackend: strauto_ignore: boolGeneral settingsdispatcher_result_backendDefault: No result backend enabled by default.The backend used to store subtask info. Can be one of the following:redis: UseRedisto store the results. SeeRedis backend settings.dispatcher_batch_sizeDefault: 100The batch size of subtask dispatching, or the result retrieving.dispatcher_poll_sizeDefault: 100The queue size of subtasks in result retrieving.celery-dispatcherput the unfinished subtasks into a queue, polling it continuously for completion of subtasks.dispatcher_poll_timeoutDefault: 10The default timeout for polling subtasks.dispatcher_subtask_timeoutDefault: 3600The default timeout in seconds beforecelery-dispatchergives up retrieving the result of each subtask.dispatcher_failure_on_subtask_timeoutDefault: FalseWhether raise exception when subtask timeout.dispatcher_failure_on_subtask_exceptionDefault: FalseWhether raise exception from subtask.dispatcher_progress_update_frequencyDefault: 1The number of steps to record.celery-dispatcheronly actually store the updated progress in the background at most every N steps.
Package: celery-executor
Celery ExecutorAconcurrent.futures.Executorimplementation using Celery as backendFree software: Apache Software License 2.0Documentation:https://celery-executor.readthedocs.io.FeaturesThe package provides aCeleryExecutorimplementing the interface ofconcurrent.futures.Executor>>>fromcelery_executor.executorsimportCeleryExecutor>>>executor=CeleryExecutor()>>>forresultinexecutor.map(str.upper,['one','two','three']):...print(result)ONETWOTHREEBeware that theExecutor.map()interface can yield the results out of order,
if later ones got to finish first.CaveatsThis executor frees the developer to the burden of mark every single task
function with the Celery decorators, and to import such tasks on the Worker
beforehand. But does not frees from sending the code to the Worker.The function sent toCeleryExecutor.map()should be pickable on the client
(caller of.map()or.submit()) and should be unpickable on the Celery
Worker handling the "Task" sent. Is not possible to send lambdas for example.As Celery assumes that is to the developer to put the needed code on the Worker,
be sure that the function/partial code sent toCeleryExecutorto exist on the
Worker.To Be DoneDocument theCeleryExecutor.__init__()nonstandard extra optionspredelay,postdelayandapplyasync_kwargs.Test behaviours of canceling a Task when canceling a FutureTest behaviours of shutting down executors and trying to send new tasksFind a way to test the RUNNING state of Celery Tasks, as its events are not propagated by the test worker Celery providesCreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.=======
History0.1.0 (2018-05-25)First release on PyPI.
Package: celery-farmer
Celery FarmerFarmer will monitor how the Celery cluster is behaving.UsageRequirementsPython, versions 3.7, 3.8, 3.9, and 3.10 are supportedCelery>=4.0, other versions may not workPip or PipenvInstallInstall package with pipenv:pipenv install celery-farmerOr pip:pip install celery-farmerRunRun celery-farmer with shell command:celery-farmerConfigurationDevelopmentRequirementsPipenvClone repository and install development requirements with:pipenv install --devRunning testsRun all tests, typecheck and linter with:pipenv run all-testsRun only typecheck with:pipenv run typecheckRun only tests with:pipenv run testOr invokepytestdirectly:pytestRun only linter with:pipenv run lintOr invokeflakedirectly:flake8
Package: celery-graceful-stop
# celery-graceful-stop[Celery](http://celeryproject.org) plugin provides ability of graceful worker stopping.# Problem
Production deployment of long running tasks require worker to be stopped gracefully. Unfortunately celery got different behaviour:ReceivingSIGTERMsignal byceleryresults startingWarm shutdownprocedure. Due this procedure,inspectandcontrolcommands become unavailable.More than that, all tasks are terminated forcely by the secondSIGTERMwith theCold shutdownprocedure.This module provides more consistent approach to this problem, itoverridesSIGTERMreceiver to prevent defaultWarm shutdownandCold shutdownworker behaviour,forcesinspectandcontrolcommands to be working even afterSIGTERMsignal received.# Installation & Setup` pip installgit+https://github.com/MnogoByte/celery-graceful-stop.git`Append yourproj/celery.pyfile containgappinstance with the following lines.`python import celery_graceful_stop celery_graceful_stop.register(app) `# SettingsCELERY_GRACEFUL_STOP(boolean). Controls graceful stop function. (Trueby default)# Using with systemdDefine 1 service per each worker you got (systemd require only one masterpidfor restart).Provide/etc/conf.d/celery_<service_name>configuration file for each worker.Add [celery@.service](systemd/celery@.service) file into your system.Register your service withsystemctl enable celery@<service_name>.# LimitationsThis module disablespool_shrink,pool_grow,autoscale,pool_reload,add_consumer,cancel_consumercontrol commands after receivingSIGTERMsignal. Actually, you’ll does not need to call them when the worker is shutting down, because worker will not starts new tasks.This module has been tested only with celery 3.1 with pool=prefork.# Author[Antonov Mikhail](https://github.com/atin65536)# LicenseBSD - 3
Package: celery-graphite
Celery GraphiteUtility for pushing celery metrics into graphiteAvaliable metrics:Number of tasks in each state[prefix.]taskname.TASK_STATE = countNumber of workers alive[prefix.]workers.alive = countEventsAlso ifgraphite-http-urlprovided task-failed events
will be pushed as graphite events with only exception name by default
or ifverbose-exceptionspecified complete task info provided.
Be aware of long exception args and kwargs.Usage:usage: celery_graphite [-h] [--config CONFIG] [--broker BROKER]
                       [--graphite GRAPHITE] [--graphite-port GRAPHITE_PORT]
                       [--graphite-retention GRAPHITE_RETENTION]
                       [--graphite-http-url GRAPHITE_HTTP_URL] [--freq FREQ]
                       [--graphite-prefix GRAPHITE_PREFIX]
                       [--verbose-exception] [--graphite-tag GRAPHITE_TAG]
                       [--verbose]

Celery graphite monitor.

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG       Config file. Must be valid python module.
  --broker BROKER       Celery broker url. (e.g. redis://localhost:6379/0)
  --graphite GRAPHITE   Graphite server url for pushing metrics.
  --graphite-port GRAPHITE_PORT
                        Graphite server port for pushing metrics.
  --graphite-retention GRAPHITE_RETENTION
                        Number of metrics for store before batch pushing.
  --graphite-http-url GRAPHITE_HTTP_URL
                        Graphite http url for pushing exception events.
  --freq FREQ           Frequency for capturing metrics.
  --graphite-prefix GRAPHITE_PREFIX
                        Prefix for graphite path.
  --verbose-exception   Export exception more verbose with args and kwargs.
  --graphite-tag GRAPHITE_TAG
                        Additional tag added to graphite events.
  --verbose, -v         Verbosity level.or any config parameter can be set via variable in file passed to --config parameter.
Should valid python module.
Package: celery-haystack
===============celery-haystack===============.. image:: https://secure.travis-ci.org/django-haystack/celery-haystack.png?branch=develop:alt: Build Status:target: http://travis-ci.org/django-haystack/celery-haystackThis Django app allows you to utilize Celery for automatically updating anddeleting objects in a Haystack_ search index.Requirements------------* Django 1.8+* Haystack_ `2.X`_* Celery_ 3.XYou also need to install your choice of one of the supported search enginesfor Haystack and one of the supported backends for Celery... _Haystack: http://haystacksearch.org.. _Celery: http://www.celeryproject.orgInstallation------------Use your favorite Python package manager to install the app from PyPI, e.g.::pip install celery-haystackFor Django < 1.9 you need to install and configure `django-transaction-hooks`_ -- an app thatbrings transaction commit hooks to Django... _django-transaction-hooks: https://github.com/carljm/django-transaction-hooksUsage-----1. Add ``'celery_haystack'`` to the ``INSTALLED_APPS`` setting.. code:: pythonINSTALLED_APPS = [# ..'celery_haystack',]2. Enable the celery-haystack signal processor in the settings.. code:: pythonHAYSTACK_SIGNAL_PROCESSOR = 'celery_haystack.signals.CelerySignalProcessor'3. Alter all of your ``SearchIndex`` subclasses to inherit from``celery_haystack.indexes.CelerySearchIndex`` and``haystack.indexes.Indexable``.. code:: pythonfrom haystack import indexesfrom celery_haystack.indexes import CelerySearchIndexfrom myapp.models import Noteclass NoteIndex(CelerySearchIndex, indexes.Indexable):text = indexes.CharField(document=True, model_attr='content')def get_model(self):return Note4. Ensure your Celery instance is running.Thanks------This app is a blatant rip-off of Daniel Lindsley's queued_search_app but uses Ask Solem Hoel's Celery_ instead of the equally awesomequeues_ library by Matt Croyden... _queued_search: https://github.com/toastdriven/queued_search/.. _Celery: http://celeryproject.org/.. _queues: http://code.google.com/p/queues/Issues------Please use the `Github issue tracker`_ for any bug reports or featurerequests... _`Github issue tracker`: https://github.com/django-haystack/celery-haystack/issues
Package: celery-haystack-ng
This Django app allows you to utilize Celery for automatically updating and
deleting objects in aHaystacksearch index.Forked from the originalcelery-haystack.RequirementsDjango 2.1+Haystack2.X+Celery4.X+You also need to install your choice of one of the supported search engines
for Haystack and one of the supported backends for Celery.InstallationUse your favorite Python package manager to install the app from PyPI, e.g.:pip install celery-haystack-ngFor Django < 1.9 you need to install and configuredjango-transaction-hooks– an app that
brings transaction commit hooks to Django.UsageAdd'celery_haystack'to theINSTALLED_APPSsettingINSTALLED_APPS=[# ..'celery_haystack',]Enable the celery-haystack signal processor in the settingsHAYSTACK_SIGNAL_PROCESSOR='celery_haystack.signals.CelerySignalProcessor'Alter all of yourSearchIndexsubclasses to inherit fromcelery_haystack.indexes.CelerySearchIndexandhaystack.indexes.Indexablefromhaystackimportindexesfromcelery_haystack.indexesimportCelerySearchIndexfrommyapp.modelsimportNoteclassNoteIndex(CelerySearchIndex,indexes.Indexable):text=indexes.CharField(document=True,model_attr='content')defget_model(self):returnNoteEnsure your Celery instance is running.ThanksThis app is a blatant rip-off of Daniel Lindsley’squeued_searchapp but uses Ask Solem Hoel’sCeleryinstead of the equally awesomequeueslibrary by Matt Croyden.IssuesPlease use theEduGit issue trackerfor any bug reports or feature
requests.
Package: celery-jalalicrontab
JalaliCrontabis aCelery schedulethat works withJalali Calendar.Getting StartedInstall with pip:pip install celery-jalalicrontabYou can use JalaliCrontab just like celery crontab. In the example bellow,
task will be run every minutes in “30” of “dey” month:@app.on_after_configure.connectdefsetup_periodic_tasks(sender,**kwargs):sender.add_periodic_task(JalaliCrontab(day_of_month=30,month_of_year=10),test.s('Happy my birthday!'),)And also you can use it alongside RedisBeater by extendingJalaliCrontabandjalalidatetimethen defineencode_beatermethod for them:classMyjalalidatetime(jalalidatetime):defencode_beater(self):ifself.tzinfoisNone:timezone='UTC'elifself.tzinfo.zoneisNone:timezone=self.tzinfo.utcoffset(None).total_seconds()else:timezone=self.tzinfo.zonedatetime_obj=self.togregorian()return{'year':datetime_obj.year,'month':datetime_obj.month,'day':datetime_obj.day,'hour':datetime_obj.hour,'minute':datetime_obj.minute,'second':datetime_obj.second,'microsecond':datetime_obj.microsecond,'timezone':timezone,}classMyJalaliCrontab(JalaliCrontab):defencode_beater(self):return{'minute':self._orig_minute,'hour':self._orig_hour,'day_of_week':self._orig_day_of_week,'day_of_month':self._orig_day_of_month,'month_of_year':self._orig_month_of_year,}DevelopmentJalaliCrontab is available onGitHubOnce you have the source you can run the tests with the following commands:pip install -r requirements.dev.txt
py.test tests
Package: celery-local-exporter
# Celery-Local-ExporterA Prometheus metrics exporter for Celery which lives directly in the celery processes. Unlike other Celery exporters, this does not require use of the events system or a result backend.## Quick StartTo install:` pip installcelery-local-exporter`To set up the exporter, in your celeryconfig.py or other settings file:` from celery_exporter import CeleryExporter CeleryExporter.install() `Thepreforkconcurrency pool is not supported at this time, I recommend using-P threads.
Package: celery-logger
Celery Loggercelery-logger is a python library for loggingceleryevents such as tasks received, tasks failed/succeeded and tasks retried, along with task args.FeaturesSimple and flexible task logsMultiple deployment options (docker, virtual machines)Integration possibilities: ELK stack and AWS cloudwatch for exampleHow does it work?Celery-logger connects with your message broker (such as redis, rabbitMQ or SQS) and logs the tasks on a file or stdout.This way you can easily have multiple celery workers and see your logs in a single place.Sample projectWe provided a sample project for playing around and seeing how it works.Getting started:# navigate to the sample-project folder
$ cd sample-project
# Build and start the containers:
$ docker-compose upQuerying logsMake sure you are in the sample-project folderCalling a few tasks$ docker-compose exec celeryd python call_tasks.py

Tasks have been called!
Run `docker-compose logs -f celery-logger` to see the logger in action.Taking a look in all events$ docker-compose logs celery-loggerSearching for failed tasks:$ docker-compose logs celery-logger | grep task-failedSearching for a specific task:$ docker-compose logs celery-logger | grep "app.add"Searching for a specific task and args:$ docker-compose logs celery-logger | grep "app.add" | grep "(6, 1)"Searching for a task id$   docker-compose logs celery-logger | grep 20925a8c-03f7-4bd7-b3dd-24e2bc9e26e2InstallationUse the package managerpipto install celery-logger.pipinstallceleryloggerUsageThe executable will be available in the path ascelery-logger:# Display the help text$celery-logger-h# Start logging from a redis broker$celery-logger--celery-brokerredis://redis:6379/0AuthorsSergio Oliveira-Initial work-SeocamThiago Ferreira-Improvements, documentation and current maintainer-thiagoferreirawAndré Girol-Packaging and distribution-Girol
Package: celery-longterm-scheduler
Schedules celery tasks to run in the potentially far future, using a separate
storage backend (currently only redis is supported) in combination with a
cronjob.UsageConfigure the storage by adding a setting likelongterm_scheduler_backend ='redis://localhost:6739/1'to your celery configuration.
(The storage also respects the built-in celery configuration settingsredis_socket_timeout,redis_socket_connect_timeoutandredis_max_connections.)Configure your celery app to use a customized task classMYCELERY = celery.Celery(task_cls=celery_longterm_scheduler.Task)Set up a cronjob to runcelery longterm_scheduler(e.g. every 5 minutes)Now you can schedule your tasks by callingmytask.apply_async(args, kwargs, eta=datetime)as normal. This returns
a normalAsyncResultobject, but only reading the.idis supported;
any other methods or properties may fail explictly or implicitly.You can completely delete a scheduled job by callingcelery_longterm_scheduler.get_scheduler(MYCELERY).revoke('mytaskid')(we cannot hook into the celery built-inAsyncResult.revoke(),
unfortunately).revoke()returns True on success and False if the given
task cannot be found in the storage backend (e.g. because it has already come
due and been executed).Instead of sending a normal job to the celery broker (with added timing
information), this creates a job entry in the scheduler storage backend. The
cronjob then periodically checks the storage for any jobs that are due, and
only then sends a normal celery job to the broker.RationaleWhy not use the celery built-inapply_async(eta=)? Because you cannot ever
really delete a pending job.AsyncResult('mytaskid').revoke()can only add
the task ID to the statedb, where it has to stay _forever_ so the job is
recognized as revoked. For jobs that are scheduled to run in 6 months time or
later, this would create an unmanageable, ever-growing statedb.Why not use celerybeat? Because it is built for periodic jobs, and we need
single-shot jobs. And then there’s not much to gain from the celerybeat
implementation, especially since we want to use redis as storage (since we’re
already using that as broker and result backend).ImplementationRedis schemacelery_longterm_scheduler assumes that it talks to a dedicated redis database.
It creates an entry per scheduled job usingSET jobidjob-configuration(job-configuration is serialized with JSON) and uses a single sorted set namedscheduled_task_id_by_timethat contains the jobids scored by the unix
timestamp (UTC) when they are due.Run testsUsingtoxandpy.test. Maybe installtox(e.g. viapip install tox)
and then simply runtox.For the integration tests you need to have the redis binary installed (tests
starttheir own server).celery_longterm_scheduler changes1.3.0 (2024-01-08)Also supportrediss://URLs1.2.0 (2022-06-23)Update to celery-5.x1.1.2 (2020-05-27)Add bw-compat so py3 can read py2-serialized tasks1.1.1 (2019-12-19)Update to current redis client library version 3.x1.1.0 (2019-11-28)Make Python-3 compatible.1.0.1 (2018-01-17)Don’t try to schedule onapply_async(eta=None)calls1.0.0 (2017-09-29)Initial release
Package: celeryman
=====================================================================Celeryman=====================================================================Celeryman is a management application for celery async tasks on django.It can manage created async tasks or you can create async task with ManagedTask or CeleryTask models.Task cannot be created with the same arguments at the same time.Installation============on Pypi::pip install celerymanon Github::git clone git@github.com:ahmetkotan/celeryman.gitcd celerymanpython setup.py installSettings============After celery integration is completed, just add the celeryman app to `INSTALLED_APPS`.::INSTALLED_APPS = [...'celeryman',...]About=====When celery service is run, Celeryman will discover tasks and save to database as CeleryTask object.If you create async task with `apply_async()`, `delay()` etc. methods, Celeryman will create ManagedTask object when task is start.::timer_task.apply_async((10,))Usage=====Async task with ManagedTask and CeleryTask model.::m = ManagedTask.objects.create(task_name='timer_task', celery_task_args=[10])m.set_task_args([10]) # if you don't use celery_task_args when created object, you can set with this method.m.start()m.stop()Or use::c = CeleryTask.objects.get(task_name='timer_task')c.set_task_args([10])managed_task = c.start()managed_task.stop()Admin Panel===========To usable the async tasks:http://localhost:8000/admin/celeryman/celerytask/To view the created async tasks and to create new async task:http://localhost:8000/admin/celeryman/managedtask/
Package: celery-mock
celery-mock allows you to mock celery task and then run them when you wantRationaleSometimes a celery task retries itself to wait for some event or model to change.
This is hard to test because celery tasks run (and retry) inline in tests.
Now you can choose when to run your tasks.Supported versionsPython 3 supportCelery 3.1.x and 4.0.x supportHow to installpipinstallcelery-mockHow to usefromcelery_mockimporttask_mockfromdjango.testimportTestCase,ClientfrommyappimportdummyviewclassUsersTestCase(TestCase):deftest_create_user(self):client=Client()client.post('/api/users/',data={'username':'konrad')# runs tasks inlinewithtask_mock():client.post('/api/users/',data={'username':'konrad')# no tasks started yet# all tasks ran herewithtask_mock('myapp.post_user_create_task'):client.post('/api/users/',data={'username':'konrad')# all tasks started execept myapp.post_user_create_task# myapp.post_user_create_task started here# you can use task_mock manually:tmock=task_mock().start()client.post('/api/users/',data={'username':'konrad')# no tasks started yettomock.stop()# all tasks ran here
Package: celerymon
Version:1.0.3InstallationYou can installcelerymoneither via the Python Package Index (PyPI)
or from source.To install usingpip,:$ pip install celerymonTo install usingeasy_install,:$ easy_install celerymonDownloading and installing from sourceDownload the latest version ofcelerymonfromhttp://pypi.python.org/pypi/celerymon/You can install it by doing the following,:$ tar xvfz celerymon-0.0.0.tar.gz
$ cd celerymon-0.0.0
$ python setup.py build
# python setup.py install # as rootUsing the development versionYou can clone the repository by doing the following:$ git clone git://github.com/celery/celerymon.gitUsageRunning the monitorStart celery with the--eventsoption on, so celery sends events for
celerymon to capture:$ python manage.py celeryd -ERun the monitor server:$ python manage.py celerymonHowever, in production you probably want to run the monitor in the
background, as a daemon:$ python manage.py celerymon --detachFor a complete listing of the command line arguments available, with a short
description, you can use the help command:$ python manage.py help celerymonNow you can visit the webserver celerymon starts by going to:http://localhost:8989Mailing listFor discussions about the usage, development, and future of celery,
please join thecelery-usersmailing list.IRCCome chat with us on IRC. The#celerychannel is located at theFreenodenetwork.Bug trackerIf you have any suggestions, bug reports or annoyances please report them
to our issue tracker athttp://github.com/celery/celerymon/issues/ContributingDevelopment ofcelerymonhappens at Github:http://github.com/celery/celerymonYou are highly encouraged to participate in the development
ofcelerymon. If you don’t like Github (for some reason) you’re welcome
to send regular patches.LicenseThis software is licensed under theNew BSD License. See theLICENSEfile in the top distribution directory for the full license text.