Package: djangoceleryrpc
django-celery-rpcRemote access from one system to models and functions of other one using Celery machinery.Relies on three outstanding python projects:CeleryDjango Rest FrameworkDjangoMain featuresClient and server are designed to:filter models with Django ORM lookups, Q-objects and excludes;change model state (create, update, update or create, delete);change model state in bulk mode (more than one object per request);atomic get-set model state with bulk mode support;call function;client does not require Django;InstallationInstall client:pipinstalldjangoceleryrpcInstall server:pipinstalldjangoceleryrpc[server]Basic ConfigurationDefault configuration ofdjango-celery-rpcmust be overridden in settings.py byCELERY_RPC_CONFIG.
TheCELERY_RPC_CONFIGis a dict which must contains at least two keys:BROKER_URLandCELERY_RESULT_BACKEND.
Any Celery config params also permitted
(seeConfiguration and defaults)serverspansetting.py:# minimal required configurationCELERY_RPC_CONFIG={'broker_url':'amqp://guest:guest@rabbitmq:5672//','result_backend':'redis://redis:6379/0',}servereggssetting.py:# alternate request queue and routing keyCELERY_RPC_CONFIG={'broker_url':'amqp://guest:guest@rabbitmq:5672/','result_backend':'amqp://guest:guest@rabbitmq:5672/','task_default_queue':'celery_rpc.requests.alter_queue','task_default_routing_key':'celery_rpc.alter_routing_key'}clientsetting.py:# this settings will be used in clients by defaultCELERY_RPC_CONFIG={'broker_url':'amqp://guest:guest@rabbitmq:5672/','result_backend':'redis://redis:6379/0',}# 'eggs' alternative configuration will be explicitly passed to the client constructorCELERY_RPC_EGGS_CLIENT={# BROKER_URL will be used by default from section above'result_backend':'amqp://guest:guest@rabbitmq:5672/','task_default_queue':'celery_rpc.requests.alter_queue','task_default_routing_key':'celery_rpc.alter_routing_key'}*Note:client and server must share the sameBROKER_URL,RESULT_BACKEND,DEFAULT_EXCHANGE,DEFAULT_QUEUE,DEFAULT_ROUTING_KEYdifferent server must serve different request queues with different routing keys or must work with different exchanges*example.pyfromcelery_rpc.clientimportClientfromdjango.confimportsettings# create client with default settingsspan_client=Client()# create client for `eggs` servereggs_client=Client(CELERY_RPC_EGGS_CLIENT)Using clientYou can find more examples in tests.FilteringSimple filtering examplespan_client.filter('app.models:MyModel', kwargs=dict(filters={'a__exact':'a'}))Filtering with Q objectfrom django.db.models import Q
span_client.filter('app.models:MyModel', kwargs=dict(filters_Q=(Q(a='1') | Q(b='1')))Also, we can use both Q and lookupsspan_client.filter('app.models:MyModel', kwargs=dict(filters={'c__exact':'c'}, filters_Q=(Q(a='1') | Q(b='1')))Exclude supportedspan_client.filter('app.models:MyModel', kwargs=dict(exclude={'c__exact':'c'}, exclude_Q=(Q(a='1') | Q(b='1')))You can mix filters and exclude, Q-object with lookups. Try it yourself. ;)Full list of available kwargs:filters - dict of terms compatible with django lookup fields
offset - offset from which return a results
limit - max number of results
fields - list of serializer fields, which will be returned
exclude - lookups for excluding matched models
order_by - order of results (list, tuple or string),
    minus ('-') set reverse order, default = []
filters_Q - django Q-object for filtering models
exclude_Q - django Q-object for excluding matched modelsList of all MyModel objects with high priorityspan_client.filter('app.models:MyModel', high_priority=True)CreatingCreate one objectspan_client.create('apps.models:MyModel',data={"a":"a"})Bulk creatingspan_client.create('apps.models:MyModel',data=[{"a":"a"},{"a":"b"}])UpdatingUpdate one object by PK field namespan_client.update('apps.models:MyModel',data={"id":1,"a":"a"})Update one object by special alias 'pk' which matched automatically to PK fieldspan_client.update('apps.models:MyModel',data={"id":1,"a":"a"})Attention! Magic area! Update one object by any field you wishspan_client.update('apps.models:MyModel',data={"alternative_key_field":42,"a":"a"},{'identity':'alternative_key_field'})Update or create, Delete and so onAll cases are very similar. Try it you console!Full list of supported model methodsfilter- select modelscreate- create new models, raise exception if model existsupdate- update existing modelsupdate_or_create- update if exist or create newdelete- delete existing modelsgetset- set new state and return old state atomicallyAll method support options:fields- shrink result fieldsserializer_cls- fully qualified symbol name to DRF serializer class on serveridentity- field name which will be used rather than PK field (mindless forfilter)PipeIt's possible to pipeline tasks, so they will be executed in one transaction.p=span_client.pipe()p=p.create('apps.models:MyModel',data={"a":"a"})p=p.create('apps.models:MyAnotherModel',data={"b":"b"})p.run()You can pass some arguments from previous task to the next.Suppose you have those models on the serverclassMyModel(models.Model):a=models.CharField()classMyAnotherModel(models.Model):fk=models.ForeignKey(MyModel)b=models.CharField()You need to create instance of MyModel and instance of MyAnotherModel which reffers to MyModelp=span_client.pipe()p=p.create('apps.models:MyModel',data={"a":"a"})p=p.translate({"fk":"id"},defaults={"b":"b"})p=p.create('apps.models:MyAnotherModel')p.run()In this example thetranslatetask:take result of the previouscreatetaskextract value of "id" field from itadd this value to "defaults" by key "fk"After that nextcreatetask takes result oftranslateas input dataAdd/delete m2m relationsLets take such models:classMyModel(models.Model):str=models.CharField()classMyManyToManyModel(models.Model):m2m=models.ManyToManyField(MyModel,null=True)Add relation between existing objectsmy_models=span_client.create('apps.models:MyModel',[{'str':'monthy'},{'str':'python'}])m2m_model=span_client.create('apps.models:MyManyToManyModel',{'m2m':[my_models[0]['id']]})# Will add 'python' to m2m_model.m2m where 'monty' already isdata={'mymodel':my_models[1]['id'],'mymanytomanymodel':m2m_model['id']}through=span_client.create('apps.models:MyManyToManyModel.m2m.through',data)And then delete some of existing relations# Next `pipe` will eliminate all relations where `mymodel__str` equals 'monty'p=span_client.pipe()p=p.filter('apps.models:MyManyToManyModel.m2m.through',{'mymodel__str':'monthy'})p=p.delete('apps.models:MyManyToManyModel.m2m.through')p.run()Run server instanceceleryworker-Acelery_rpc.appServer with support task consuming prioritizationcelerymultistart2-Acelery_rpc.app-Q:1celery_rpc.requests.high_priorityNote, you must replace 'celery_rpc.request' with actual value of config paramCELERY_DEFAULT_QUEUECommand will start two instances. First instance will consume from high priority queue only. Second instance will serve both queues.For daemonization seeRunning the worker as a daemonRun testspythondjango-celery-rpc/celery_rpc/runtests/runtests.pyMore ConfigurationOverriding base task classOVERRIDE_BASE_TASKS={'ModelTask':'package.module.MyModelTask','ModelChangeTask':'package.module.MyModelChangeTask','FunctionTask':'package.module.MyFunctionTask'}Supported class names:ModelTask,ModelChangeTask,FunctionTaskHandling remote exceptions individually# Both server and clientCELERY_RPC_CONFIG['wrap_remote_errors']=TrueAfter enabling remote exception wrapping client will raise same errors happened
on the server side.
If client side has no error defined (i.e. no package installed),Client.RemoteErrorwill be raised.
Also,Client.RemoteErroris a base for all exceptions on the client side.For unknown exceptions this code is valid:try:result=rpc_client.call("remote_func")exceptrpc_client.errors.SomeUnknownErrorase:# here a stub for remote SomeUnknownError is handledprint(e.args)For known exceptions both variants work:try:result=rpc_client.call("remote_func")exceptrpc_client.errors.MultipleObjectsReturnedase:# django.core.exceptions.MultipleObjectsReturnedhandle_error(e)exceptdjango.core.exceptions.ObjectDoesNotExistase:# django.core.exceptions.ObjectDoesNotExisthandle_error(e)If original exception hierarchy is needed:SomeBaseError=rpc_client.errors.SomeBaseErrorDerivedError=rpc_client.errors.subclass(SomeBaseError,"DerivedError")TODOSet default non-generic model serializer.Test support for RPC result backend from Celery.Token auth and permissions support (like DRF).Resource map and strict mode....AcknowledgementsThanks for all who contributing to this project:https://github.com/voron3xhttps://github.com/dtrsthttps://github.com/anatoliy-larinhttps://github.com/bourivouhhttps://github.com/tumb1er