Saravanan666/pypi-org huggingface dataset:
Package: celery-api
Given the celery instance, it inspects all available celery workers to get
the information about the queues they serve and tasks they know about.Then it creates a chain of attributes allowing to execute any task asqueue_name.full_task_name.delay.With help of this class you can turn your Celery installation to a set of
independent modules, each of which “exposes” its own “Celery API”.To make it more clear, the analogy with a random HTTP-based API available
athttp://example.com/users/get?email=john@example.comcan be like:Celery object (including broker URL, result backend settings, etc) is the
analogue of the protocol (http://)Queue name is the analogue of the hostname (example.com)Task name is the analogue of the URL path (/users/get)Task parameters the the analogue the querystring (?email=john@example.com)Usage example.If we have a Celery installation with two queues:
“download” (knows how to execute “downloader.download_url” task) and
“parse” (knows how to execute “parser.parse_html”), we can instantiate
API and work with it the following way:>>>api=celery_api.CeleryApi(celery)>>>html_page=api.download.downloader.download_html.delay('http://example.com').get()>>>html_tree=api.parse.parser.parse_html.delay(html_page).get()NoteEnsure that workers are up and available from clients for inspection.
You may re-discover your installation after object creation by executingapi._discover().
Package: celery-aws-xray-sdk-extension
celery-aws-xray-sdk-extensionCelery signal handlers that integrate Celery task lifecycle with AWS X-Ray tracing.There's a tiny example in the directoryexample.InstallationYou can install it easily withpip:pip install celery-aws-xray-sdk-extension. For latest version visitPyPI.SetupThis guide doesn't cover setting up AWS X-Ray SDK for Python or AWS X-Ray daemon. It's expected that you've already got some experience with it. If you don't have any experience with AWS X-Ray, please visitAmazon documentation.You have to haveCelery signalsenabled.Connect Celery signals to signal handlers fromcelery_aws_xray_sdk_extensionmodule in your Celery setup. Example code is below this list.You're good to go!Connecting handlers to Celery signalssignals.after_task_publish.connect(xray_after_task_publish)signals.before_task_publish.connect(xray_before_task_publish)signals.task_failure.connect(xray_task_failure)signals.task_postrun.connect(xray_task_postrun)signals.task_prerun.connect(xray_task_prerun)DevelopmentBefore making any changes, please prepare your environment (no global changes are made!) withmake setup-dev. Feel free to check what the target does in theMakefile.This repository usesConventional Commits.
Package: celery-batches
Celery BatchesCelery Batches provides aTaskclass that allows processing of multiple
Celery task calls together as a list. The buffer of tasks calls is flushed on a
timer and based on the number of queued tasks.Some potential use-cases for batching of task calls include:De-duplicating tasks.Accumulating / only handling the latest task with similar arguments.Bulk inserting / updating of data.Tasks with expensive setup that can run across a range of arguments.What do I need?celery-batches version runs on,Python (3.8, 3.9, 3.10, 3.11)PyPy3 (7.6)And is tested with Celery >= 5.0.If you’re running an older version of Python, you need to be running
an older version of celery-batches:Python 2.7: celery-batches 0.3.Python 3.4: celery-batches 0.2.Python 3.5: celery-batches 0.3.Python 3.6: celery-batches 0.5.Python 3.7: celery-batches 0.7.If you’re running an older version of Celery, you need to be running
an older version of celery-batches:Celery < 4.0: Usecelery.contrib.batchesinstead.Celery 4.0 - 4.3: celery-batches 0.3.Celery 4.4: celery-batches 0.7.HistoryCelery Batches was distributed as part of Celery (ascelery.contrib.batches)
until Celery 4.0. This project updates the Batches code to maintain compatiblity
with newer versions of Celery and other fixes. See the Changelog for details.Additionally, this repository includes the full history of the code fromcelery.contrib.batches, but rewritten to thecelery_batches/__init__.pyfile.
Package: celerybeatmongo
# celerybeatmongoA very simple Celery Beat Scheduler that stores status information in a MongoDB database.# Install```pip install celerybeatmongo```# DetailsThis project provides a Celery Beat Scheduler that uses MongoDB to store the schedule stats (last run, run count, etc.). This is safer than using a file on your server which could be cleared or corrupted (on Heroku for example). This scheduler aims to replace the default Beat Scheduler provided with Celery. There are a few caveats though:* Only supports interval scheduling (currently, easy to add support for Crontab and Solar)* Can only pass native JSON types (dict, list, str, number) to `kwargs`, `args`, and `options` for tasks## MotivationI started this project because I run Celerybeat on Heroku and I started noticing that tasks were being dropped. This was because the default Celerybeat Scheduler (`[celery.beat.PersistentScheduler](http://docs.celeryproject.org/en/latest/reference/celery.beat.html#celery.beat.PersistentScheduler)`) uses a shelve database file that it adds the current directory. In Heroku this file was cleared and so the schedule data was not saved reliably. With the `celerybeatmongo.MongoPersistentScheduler` this is not an issue anymore.# ExamplePlease see `examples/celery_worker.py` for a simple example to just try it out.Running the example requires MongoDB and Redis (you can edit it to use another Broker instead if Redis if you want).* [Install Redis](https://redis.io/topics/quickstart)* [Install MongoDB](https://docs.mongodb.com/manual/installation/)You can run the following to start the example:1. In a seperate terminal window, start Redis:```shell$ redis-server      # start redis```2. In a seperate terminal window, start MongoDB:```shell$ mongod```3. Then start the example worker:```shell$ pip install redis$ python examples/celery_worker.py```# ContributePlease add issues or put up pull requests if you need more functionality or have suggestions. Its a new project and should be easy to contribute to!
Package: celerybeat-redis
celerybeat-redisIt’s modified from celerybeat-mongo
(https://github.com/zakird/celerybeat-mongo)This is a Celery Beat Scheduler
(http://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html)
that stores both the schedules themselves and their status information
in a backend Redis database.FeaturesFull-featured celery-beat schedulerDynamically add/remove/modify tasksSupport multiple instance by Active-Standby modelInstallationIt can be installed by installing the celerybeat-redis Python egg:# pip install celerybeat-redisAnd specifying the scheduler when running Celery Beat, e.g.$ celery beat -S celerybeatredis.schedulers.RedisSchedulerSee more details in Github:https://github.com/kongluoxing/celerybeatredis
Package: celery-beatx
Celery-BeatX is a modern fail-safe schedule for Celery.Celery-BeatX allows you to store schedule in different storages and
provides functionality to start celery-beat simultaneously at many nodes.Features:stores schedule in different storages (currently support: redis, memcached)allows to correctly run several instances of celerybeat simultaneouslySee thedocumentationfor more details.InstallThe package can be installed using:pip install celery-beatxAfter package installed you should set celery scheduler:celery_app.config_from_object({
   # ...
   'beat_scheduler': 'beatx.schedulers.Scheduler',
   'beatx_store': 'redis://127.0.0.1:6379/',
   # ...
})ContributingIf you have any valuable contribution, suggestion or idea,
please let us know as well because we will look into it.Pull requests are welcome too.
Package: celery-bungiesearch
A repository that wraps the provided Bungiesearch signal processing class
in your own celery tasks.This Django app allows you to utilize Celery for automatically updating and deleting
objects in a Bungiesearch search index.Compatible with most recent version of Bungiesearch, which is similar to Haystack,
but interacts directly with elasticsearch-dsl and therefore allows for more custom
ranking functions.Installpipinstallcelery-bungiesearchRequirementsDjango 1.4.3+Bungiesearch1.2.1+Celery3.1.18+Usage1. Add'celery_bungiesearch'toINSTALLED_APPSin settingsINSTALLED_APPS=[# ..'celery_bungiesearch',]2. AddCelerySignalProcessortosettings.BUNGIESEARCH['SIGNAL_CLASS']in settings. This will ensure that any model that’s managed by aBungiesearchManageracquireCelerySignalProcessoras the signal processor.fromcelery_bungiesearchimportCelerySignalProcessorBUNGIESEARCH={# ..'SIGNALS':{'SIGNAL_CLASS':'celery_bungiesearch.signals.CelerySignalProcessor','BUFFER_SIZE':100}}3. Add celery-bungisearch configuration variables to your settings file. The task below is the default version, but you may include your own custom classes if you desire (note that none of these environment variable are required, and can be entirely excluded from the settings file):CELERY_BUNGIESEARCH_QUEUE=NoneCELERY_BUNGIESEARCH_TASK='Your custom index task path'CELERY_BUNGIESEARCH_CUSTOM_TASK='Your custom celery task path'4. Ensure your Celery instance is running.TestingYou can run tests locally for celery-bungiesearch by simply running the commandtoxortox test. You must have an instance of elasticsearch running locally. You can also run tests using continuous integration with Travis CI (build status at the top of this README).ThanksThis application borrows liberally from Jannis Leidel’scelery-haystackand from Christopher Rabotin’sBungiesearch, which itself was the inspiration for this project.IssuesPlease submit a pull request or use theGithub issue trackerfor any bug fixes, bug reports, or feature requests.
Package: celery-client-stubs
Celery stubsThis is a small library for client side function stubs for celery.The namestubsdoes not refer to the testing aspect but rather to a proxy behavior coming from the stub / skeleton pattern known from remote method invocations. At celery, there is no need to create a skeleton, since this is handled by the middleware.The stubs approach uses thesend_taskto schedule a remote task for execution. Since the stub does not know the arguments to apply, this is the part of the blueprint method.Example## Server@app.task(name="calculate_sum")defsum(a,b):returna+b## Client# Proxy Definitionclass_CalculateSumTask(celery_client_stubs.AsyncRemoteTask):def__init__(self,celery,*args):super().__init__("calculate_sum",celery,*args)defcelery_sum(celery,a,b):return_CalculateSumTask(celery,a,b)# Definition of methodsapp=Celery(...)...defsum(a,b):returncelery_sum(app,a,b)# Usageasync_result=sum(1,2).schedule_immediately()# Wait for method to complete ...Task FactoryThe class itself provides anRemoteTaskFactory, that can be used:# DefinitionclassMyRemoteTasks(celery_client_stubs.RemoteTaskFactory):defsum(self,a,b):task=celery.AsyncRemoteTask("calculate_sum",self._celery,a,b)returntask# Usageapp=Celery(...)tasks=MyRemoteTasks(app)async_result=tasks.sum(1,2).schedule_immediately()LicensingThis library is published under BSD-3-Clause license, just like Celery is.VersioningThis library follows semantic versioning 2.0. Any breaking change will produce a new major release. Versions below 1.0 are considered to have a unstable interface.
Package: celery-cloudwatch
#Celery CloudWatchMonitor yourceleryapplication from withinAWS CloudWatch!##MetricsThe following events are tallied per task:CeleryEventSentCeleryEventStartedCeleryEventSucceededCeleryEventFailedYou can then see how many tasks/day, tasks/week etc are being completed.Also, we record the total waiting/running per task:CeleryNumWaitingCeleryNumRunningAlso, statistics on task duration are sent in the metrics:CeleryWaitingTimeCeleryProcessingTimeThese metrics are sent with all supported stats (No. Events, Sum, Max, Min), allowing you to gain insight into your task processing and match requests and capacity.#Getting StartedSet up anIAM Rolefor your instance.It must include a policy to perform 'PutMetricData', eg:{"Version":"2000-01-01","Statement":[{"Effect":"Allow","Action":["cloudwatch:PutMetricData"],"Resource":["*"]}]}(Note: Alternatively, you can set up aUserwith the same policy and provide access details that way)Install viapython-pip(and upgrade pip & boto)sudoapt-getinstall-ypython-pip
sudopipinstall--upgradepipboto# Install directlysudopipinstallcelery-cloudwatch# OR, install in a virtualenvsudoapt-getinstall-ypython-virtualenv
mkdir/var/python-envs
virtualenv/var/python-envs/ccwatchsource/var/python-envs/ccwatch/bin/activate
pipinstallcelery-cloudwatchCreate your ownboto.cfgat/etc/boto.cfg-[Credentials]
# if not using an IAM Role - provide aws key/secret
aws_access_key_id = xxx
aws_secret_access_key = yyy

[Boto]
cloudwatch_region_name = my-region
cloudwatch_region_endpoint = monitoring.my-region.amazonaws.comCreate your own config file in/etc/ccwatch.yamlccwatch:
  broker: null
  camera: celery_cloudwatch.CloudWatchCamera
  verbose: no
camera:
  frequency: 60.0
  verbose: no
cloudwatch-camera:
  dryrun: no
  namespace: celery
  tasks:
    - myapp.mytasks.taskname
    - myapp.mytasks.anothertask
    - myapp.mytasks.thirdtask
    - name: myapp.secondarytasks
      dimensions:
        task: myapp.secondarytasks
        customDim: value
    - name: myapp.tertiarytasks
      dimensions:
        task: myapp.tertiarytasks
        customDim: valueInstall upstartCreate a file/etc/init/celery-cloudwatch.conf-description "Celery CloudWatch"
author "nathan muir <ndmuir@gmail.com>"

setuid nobody
setgid nogroup

start on runlevel [234]
stop on runlevel [0156]

exec /var/python-envs/ccwatch/bin/ccwatch
respawnthen-sudoinitctlreload-configuration
sudoservicecelery-cloudwatchstartStart Celery your celery workers with the-E(orCELERY_SEND_EVENTS=1andCELERY_TRACK_STARTED=1) options, and, start celery clients withCELERY_SEND_TASK_SENT_EVENT=1All done! head over to your CloudWatch monitoring page to see the results!
Package: celery-cloudwatch-logs
This is a simple monitoring tool that intercepts the results of Celery tasks and uploads the result to AWS CloudWatch.InstallationInstall from PyPi:pipinstallcelery-cloudwatch-logsConfigurationConfiguration is done through environment variables. The following environment variables must be set in order forcelery-cloudwatchto work:AWS_CLOUDWATCH_ACCESS_KEYAWS_CLOUDWATCH_SECRET_KEYAWS_CLOUDWATCH_GROUP_NAMEOptionally, the following environment variables may be set:REDIS_URL="redis://"AWS_CLOUDWATCH_REGION="eu-west-1"The CloudWatch log group does not have to exist. If it doesn’t exists, it will be created.Running$celerycloudwatchThis requires you to have ransetup.pyor installed this package through pip. Alternatively, run:$python-mcelery_cloudwatchHow it worksCelery CloudWatch connects to your broker and monitors tasks in real time. When a task succeeded, failed was rejected or revoked, it uploads all available information about that task into a log stream on AWS CloudWatch Logs.Based on the specified log group name in theAWS_CLOUDWATCH_GROUP_NAME, a log group will be created. For each possible result, a separate log stream is created. For each task result, an entry is added to the log stream associated with the result.Known issues--brokeroncelery cloudwatchis ignored.No descriptive way to specify other brokers than Redis.All brokers supported by Celery will work, simply specify the broker URL through theREDIS_URLenvironment variable.
Package: celery-context
DocumentationThe documentation is hosted athttps://github.com/BingerYang/celery_contextInstallationadding the trace_id, task_name and task_id at celery log formatpipinstallcelery_contextUsageexample:# -*- coding: utf-8 -*-
from flask import Flask
from celery_context import Celery
from flask import request
from .. import celery_settings  # 存放celery 配置的文件
"""celery_settings
worker_log_format = '[%(trace_id)s%(task_name)s%(task_id)s%(process)s %(thread)s %(asctime)s %(pathname)s:%(lineno)s] %(levelname)s: %(message)s'
worker_task_log_format = worker_log_format
"""

config = dict(redis={"host": "*****", "port": 31320, "password": "lab@2019"})
redis_url = "redis://:{password}@{host}:{port}".format(**config["redis"])
app = Flask("example.run")
# app.config['CELERY_BROKER_URL'] = "{}/1".format(redis_url)
# app.config['CELERY_RESULT_BACKEND'] = "{}/2".format(redis_url)
celery = Celery(app=app, broker=redis_url, backend=redis_url)
celery.config_from_object(celery_settings)
celery.add_flask_content(app)


@celery.task(bind=True)
def add(self, a, b):
    print("a+ n:", a + b, self.request.content)
    return a + b


@app.route("/")
def index():
    ret = add.delay(1, 3)
    print(ret.get())
    return ret.id


if __name__ == "__main__":
    app.run()
Package: celerycontrib.sqlalchemyscheduler
=============================SQLAlchemyScheduler=============================.. image:: https://travis-ci.org/adblair/celerycontrib.sqlalchemyscheduler.svg?branch=master:target: https://travis-ci.org/adblair/celerycontrib.sqlalchemyscheduler.. image:: https://coveralls.io/repos/adblair/celerycontrib.sqlalchemyscheduler/badge.svg?branch=master&service=github:target: https://coveralls.io/github/adblair/celerycontrib.sqlalchemyscheduler?branch=master.. image:: https://img.shields.io/pypi/dm/celerycontrib.sqlalchemyscheduler.svg:target: https://pypi.python.org/pypi/celerycontrib.sqlalchemyschedulerSQLAlchemy-backed scheduler for Celery.History-------0.1.0-dev0 (2015-09-07)++++++++++++++++++* First release.
Package: celery-debounce
Celery DebounceCelery debounce without using cache.How to create debounce task?from celery_debounce.celery_debounce import DebouncedTask

@app.task(bind=True, base=DebouncedTask)
def testing_task(self, some_id):
    """
    how to call task?
    testing_task.debounce(some_id=1234)
    """
    print("from testing_task", some_id)How to call the task?testing_task.debounce(some_id=1234)With countdown (By default it's 60 seconds):testing_task.debounce(some_id=1234, countdown=10)=======
History0.1.0 (2022-10-21)First release on PyPI.
Package: celery-decorator-taskcls
Celery Decorator taskclsFree software: MIT LicenseThis package is temporary way to getapp.taskclsdecorator right now.
The main target of this package make you able to use taskcls decorator
before celery 4.5 released, and then you can remove this package importwithoutapplication code changeMore about:https://github.com/celery/celery/pull/5755Featuresimport celery_decorator_taskcls
celery_decorator_taskcls.patch_celery()

from celery import Celery
app = Celery(...)

class BaseTask:
    def __init__(self, task, **kwargs):
        self.task = task
        for key, value in kwargs.items():
            setattr(self, key, value)

    @classmethod
    def task(cls, task, **kwargs):
        instance = cls(task, **kwargs)
        return instance.main()


@app.taskcls(bind=True)
class SimpleTask(BaseTask):
    def main(self):
        ...app.taskclsdecorator behavior is the same asapp.task. You can pass it
kwargs likebind,nameor other or you can use it without kwargsYou can also pass default decorator options by nested classMetaTask:class BaseTask:
    class MetaTask:
        bind = True

    @classmethod
    def task(cls, taks, *args, **kwargs):
        ...Patching optionsBy default patcher searchCelery.taskclsattribute. If it not found, patcher
creates it. But when it exists (I belive you find it in Celery 4.5), patcher
checks its optional argumentforce, because it seems patching not required.
Callingcelery_decorator_taskcls.patch_celery(force=True)enforces
patching Celery evenCelery.taskclsexistsChangelog[0.1.4]Allow to inherit taskcls from another taskclsComment code blocks to simplify its understanding[0.1.3]Fix MetaTask inheritance[0.1.2]Update Readme[0.1.1]Add chandelogRename nested classMetatoMetaTask[0.1.0]Initial commit
Package: celery-dill-serializer
There are some issues withNameErrorwhen serialized functions depend on values defined outside of the scope of thedef.
Try putting imports and such inside the function definition, i.e.defmake_me_a_pi():importmathreturnmath.pias opposed toimportmathdefmake_me_a_pi():returnmath.piwhich would result inNameError: name 'math' is not defined.
Package: celery-director
Director is a simple and rapid framework used to manage tasks and build workflows using Celery.The objective is to make Celery easier to use by providing :a WebUI, an API and a CLI to manage and execute the workflows,a WebUI to track the tasks states,a YAML syntax used to combine tasks into workflows,the ability to periodically launch a whole workflow,and many others.See how to use Director with the quickstart and guides in thedocumentation.InstallationInstall the latest version of Director with pip (requires at leastPython 3.7):pipinstallcelery-directorUsageWrite your code in Python# tasks/orders.pyfromdirectorimporttaskfrom.utilsimportOrder,Mail@task(name="ORDER_PRODUCT")deforder_product(*args,**kwargs):order=Order(user=kwargs["payload"]["user"],product=kwargs["payload"]["product"]).save()return{"id":order.id}@task(name="SEND_MAIL")defsend_mail(*args,**kwargs):order_id=args[0]["id"]mail=Mail(title=f"Your order #{order_id}has been received",user=kwargs["payload"]["user"])mail.send()Build your workflows in YAML# workflows.ymlproduct.ORDER:tasks:-ORDER_PRODUCT-SEND_MAILRun itYou can simply test your workflow in local :$directorworkflowrunproduct.ORDER'{"user": 1234, "product": 1000}'And run it in production using the director API :$curl--header"Content-Type: application/json"\--requestPOST\--data'{"project": "product", "name": "ORDER", "payload": {"user": 1234, "product": 1000}}'\http://localhost:8000/api/workflowsRead thedocumentationto try the quickstart and see advanced usages of Celery Director.Project layout.env                # The configuration file.
workflows.yml       # The workflows definition.
tasks/
    example.py      # A file containing some tasks.
    ...             # Other files containing other tasks.Commandsdirector init [path]- Create a new project.director celery [worker|beat|flower]- Start Celery daemons.director webserver- Start the webserver.director workflow [list|show|run]- Manage your project workflows.LicenseSeehttps://github.com/ovh/celery-director/blob/master/LICENSE