Package: pyfaktory
Faktory Client Python (faktory_worker_python)This repository provides Python Client (Consumer and Producer) for theFaktorybackground job server.+--------------------+
                   |                    |
                   |     Faktory        |
                   |     Server         |
          +---->>>>|                    +>>>>----+
          |        |                    |        |
          |        |                    |        |
          |        +--------------------+        |
          |                                      |
          |                                      |
          |                                      |
          |                                      |
+-----------------------------------------------------------+     
|         .                Faktory               .          |            
|         .                Client                .          |  
|         .                                      .          |          
|   +-----------------+               +-----------------+   |
|   |                 |               |                 |   |
|   |    Producer     |               |    Consumer     |   |
|   |     pushes      |               |    (Worker)     |   |
|   |      jobs       |               |     fetches     |   |
|   |                 |               |       jobs      |   |
|   |                 |               |                 |   |
|   +-----------------+               +-----------------+   |          
|                                                           |            
+-----------------------------------------------------------+Server- the Faktory daemon which stores background jobs in queues to be processed by Workers.Client - an entity that communicates with the Faktory server using theFWP. A single client can act as both a consumer and a producer.Consumer (or Worker) - a client that fetches work units (jobs) from the server for execution.Producer - a client that issues new work units to the server.This library tries to implement theFWPas well as possible. If you notice any inconsistencies, please report them.Installationpip install pyfaktoryUsageFaktory ServerIf you have a Faktory server already running, make sure you have the correct url.# Default url for a Faktory server running locallyfaktory_server_url='tcp://localhost:7419'For the installation of faktory, please refer tothe official documentation.After installation, you can run it locally.$/usr/bin/faktoryFaktory 1.8.0You can use a password for the Faktory server via the environment variableFAKTORY_PASSWORD. Note if this value starts with a/, then it is considered a pointer to a file on the filesystem with the password. By default/etc/faktory/passwordis used.The format of the Faktory URL is as follows:tcp://:password@localhost:7419You can access theFakotry GUI.To run Faktory in production:/usr/bin/faktory -e productionFaktory in production mode requires a password by default since version 0.7.0.Faktory ClientImportpyfaktory.frompyfaktoryimportClient,Consumer,Job,ProducerA single client can act as both a consumer and a producer.client=Client(faktory_url='tcp://localhost:7419')client.connect()# Now you can use the client# At the end, disconnect the clientclient.disconnect()Client is a context manager, so you can usewithstatement.withClient(faktory_url='tcp://localhost:7419')asclient:# Use clientUseroleargument to say how you want to use the client. This argument has
three possible values: 'consumer', 'producer' or 'both'.# A client that acts as both a consumer and a producer.client=Client(faktory_url='tcp://localhost:7419',role='both')ProducerUse the client to push jobs.Push jobwithClient(faktory_url='tcp://localhost:7419',role='producer')asclient:producer=Producer(client=client)job_1=Job(jobtype='adder',args=(5,4),queue='default')producer.push(job_1)Push bulk jobsYou can push several jobs at once. There is no limit, but 1000 at a time is recommended as a best practice.withClient(faktory_url='tcp://localhost:7419',role='producer')asclient:producer=Producer(client=client)job_2=Job(jobtype='adder',args=(50,41))job_3=Job(jobtype='adder',args=(15,68))res=producer.push_bulk([job_2,job_3])Consumer (Worker)Use a worker to pull jobs from Faktory server and execute them.defadder(x,y):logging.info(f"{x}+{y}={x+y}")withClient(faktory_url='tcp://localhost:7419',role='consumer')asclient:consumer=Consumer(client=client,queues=['default'],concurrency=1)consumer.register('adder',adder)consumer.run()Usepriorityto indicates in which queue order the jobs should be fetched
first.# With strict priority, there is a risk of starvationconsumer=Consumer(client=client,queues=['critical','default','bulk'],priority='strict')# Each queue has an equal chance of being fetched firstconsumer=Consumer(client=client,queues=['critical','default','bulk'],priority='uniform')# Weights must be specifiedconsumer=Consumer(client=client,queues=['critical','default','bulk'],priority='weighted',weights=[0.6,0.3,0.1])Capture exceptions using SentryTo capture exceptions using Sentry before failling jobs
setsentry_capture_exceptionargument toTrue.consumer=Consumer(client=client,sentry_capture_exception=True)InfoYou can get various information about the server usinginfoClient method.withClient(faktory_url='tcp://localhost:7419')asclient:server_info=client.info()print(server_info)MutateA wrapper for theMutate APIto script certain repairs or migrations.⚠️ MUTATE commands can be slow and/or resource intensive. They should not be used as part of your application logic.frompyfaktoryimportClient,JobFilter,MutateOperationclient=Client(faktory_url='tcp://localhost:7419')client.connect()# Find all scheduled jobs with type `QuickbooksSyncJob` and discard themop=MutateOperation(cmd='discard',target='scheduled',filter=JobFilter(jobtype="QuickbooksSyncJob"))client.mutate(op)# Clear the Retry queue completelyop=MutateOperation(cmd='discard',target='retries',filter=JobFilter(regexp="*"))client.mutate(op)# Clear the Retry queue completelyop=MutateOperation(cmd='discard',target='retries')client.mutate(op)# Send a two specific JIDs in the Retry queue to the Dead queueop=MutateOperation(cmd='kill',target='retries',filter=JobFilter(jobtype="QuickbooksSyncJob",jids=["123456789","abcdefgh"]))client.mutate(op)# Enqueue all retries with a first argument matching "bob"op=MutateOperation(cmd='requeue',target='retries',filter=JobFilter(regexp="*\"args\":[\"bob\"*"))client.mutate(op)client.disconnect()QueuesPausingQueues may be paused (no job can be fetched from them while paused) or unpaused
(resume fetching).# Pause a list of queuesclient.queue_pause(queues=['bulk','another_queue'])# Pause all queuesclient.queue_pause(all_queues=True)# Unpause a list of queuesclient.queue_unpause(queues=['bulk'])# Unpause all queuesclient.queue_unpause(all_queues=True)RemoveQueues can be removed which deletes all jobs within them. It does not stop
currently executing jobs from those queues.# Remove a list of queuesclient.queue_remove(queues=['bulk'])# Remove all queuesclient.queue_remove(all_queues=True)Batch (untested)Refer todocumentation.frompyfaktoryimportBatch,Client,Job,Producer,TargetJobclient=Client(faktory_url='tcp://localhost:7419')client.connect()producer=Producer(client=client)batch=Batch(description="An optional description for the Web UI",success=TargetJob(jobtype="MySuccessCallback",args=[123],queue="critical"),complete=TargetJob(jobtype="MyCompleteCallback",args=['aa'],queue="critical"))# Create a new batchresp=producer.batch_new(batch)# Push as many jobs as necessary for the batch# You may nest batches# The initial batch data has a TTL of 30 minutes and will expire if batch is not commitedproducer.push(Job(jobtype='SomeJob',args=(5,4),custom={"bid":bid}))producer.push(Job(jobtype='SomeOtherJob',args=(0,15),custom={"bid":bid}))# Commit the batchproducer.batch_commit(bid)client.disconnect()Usebatch_opento open a created batch.producer.batch_open(bid)Useparent_bidfor child batch.child_batch=Batch(parent_bid=bid,description="An optional description for the Web UI",success=TargetJob(jobtype="MySuccessCallback",args=[123],queue="critical"),complete=TargetJob(jobtype="MyCompleteCallback",args=['aa'],queue="critical"))Custom commandIf you want to use a Faktory command that is not yet implemented in this client library, you can send custom commands.frompyfaktoryimportClientmy_command='INFO\r\n'withClient(faktory_url='tcp://localhost:7419')asclient:resp=client._send_and_receive(my_command)print(resp)ExampleFind examples in./examples.Start the Faktory server./usr/bin/faktoryLaunch a producer.python examples/fproducer.pyLaunch a consumer.python examples/fconsumer.pyLook at what is happening in the logs and in theFaktory Web UI.ContributeIssuesIf you encounter a problem, please report it.In addition to the description of your problem, report the server and client
versions.pyfaktory.__version__/usr/bin/faktory -vReproduce your problem while increasing the level of debugging for both the
server and the client, and report the logs.importlogginglogging.basicConfig(format='%(asctime)s%(levelname)-8s%(message)s',level=logging.DEBUG,datefmt='%Y-%m-%d%H:%M:%S')/usr/bin/faktory -l debugPRsPlease, feel free to create pull requests.LicenseSee theLICENSEfile for license rights and limitations (MIT).