Package: zc.async
What is it?Thezc.asyncpackage providesan easy-to-use Python tool that schedules
work persistently and reliably across multiple processes and machines.For instance…Web apps: maybe your web application lets users request the creation of a
large PDF, or some other expensive task.Postponed work: maybe you have a job that needs to be done at a certain time,
not right now.Parallel processing: maybe you have a long-running problem that can be made
to complete faster by splitting it up into discrete parts, each performed in
parallel, across multiple machines.Serial processing: maybe you want to decompose and serialize a job.High-level features include the following:easy to use;flexible configuration, changeable dynamically in production;reliable;supports high availability;good debugging tools;well-tested; andfriendly to testing.While developed as part of the Zope project, zc.async can be used stand-alone.How does it work?The system uses the Zope Object Database (ZODB), a transactional, pickle-based
Python object database, for communication and coordination among participating
processes.zc.async participants can each run in their own process, or share a process
(run in threads) with other code.The Twisted framework supplies some code (failures and reactor implementations,
primarily) and some concepts to the package.Where can I read more?Quickstarts and in-depth documentation are available in the package and in
thenew and exciting on-line documentation.Changes1.5.4 (2011-03-03)Rearrange ftesting.setUp to avoid provoking a DemoStorage bug
present in ZODB <= 3.9.3.Resolved the following testing issues: signal handlers weren’t cleaned
up properly in some tests, Twisted was leaking file descriptors during
ftesting tearDown (http://twistedmatrix.com/trac/ticket/3063), and
the twisted.txt regression test was not repeatable due to
Twisted’s recalcitrance when it comes to stopping and subsequently
starting a reactor instance.Fix two undefined variables that could trigger exceptions in corner
cases.1.5.3 (2009-11-15)Made zc.async.subscribers.ThreadedDispatcherInstaller keep track of signal
handlers it installs in a module global “signal_handlers.”Made zc.async.ftesting.tearDown restore the signal handlers that were
replaced by ThreadedDispatcherInstaller.Fix a bug in zc.async.ftesting.setUp and zc.async.testing.print_logs
which would result in the default argument for log_file becoming
“fixated” with an incorrect value across tests.Make the ftesting.txt test exercise the ‘zc.async’ logger in
addition to ‘zc.async.event’.zc.async.utils.dt_to_long coerces return value to long (test pass on 64-bit
Python).Tests pass on Python 2.6The callable of a zc.async.job.Job (or one of its subclasses) can be a
method on the Job itself.1.5.2 (2009-07-22)Fix a bug where zc.async.testing._datetime.now did not accept the same
keyword arguments as datetime.datetime, added tests.Fix a bug where zc.async.testing._datetime.astimezone did not accept the same
keyword arguments as datetime.datetime, added tests.1.5.1 (2008-10-13)Add a performance optimization withisinstancebeforeprovidedBy.Add support for filters toAgentInstaller.Fix a bug which caused a condition to be always true inmonitor.Encoder.1.5.0 (2008-09-21)Documentation improvements.  Converted documentation into Sphinx system.Made “other” commit errors for theRetryCommonForeverretry policy have
an incremental backoff.  By default, this starts at 0 seconds, and increments
by a second to a maximum of 60 seconds.Work around a memory leak in zope.i18nmessageid
(https://bugs.launchpad.net/zope3/+bug/257657).  The change should be
backward-compatible.  It also will produce slightly smaller pickles for Jobs,
but that was really not a particular goal.Added zc.async.partial.Partial for backward compatibility purposes.Fix support for Twisted installed reactor.Fix retry behavior for parallel and serial jobsTweaked the uuid.txt to mention zdaemon/supervisor rather than Zope 3.Fixed some bugs in egg creation.Changed quotas to not use a container that has conflict resolution, since
these values should be a strict maximum.We only want to claim a job if we are activated.   Make the agent check theactivatedanddeadattributes of the parent dispatcher before
claiming.When activating, also clean out jobs from the dispatcher’s agents, just as
with deactivating.  This should protect from unusual race conditions in
which the dispatcher got a job after being deactivated.Change dispatcher to ping before claiming jobs.when a ping reactivates a dispatcher, use new methodreactivaterather
thanactivate.  This fires a newDispatcherReactivatedevent.It’s still theoretically possible (for instance, with a
badly-behaved long commit that causes a sibling to believe that the
process is dead) that an async worker process would be working on a
job that it shouldn’t be.  For instance, the job has been taken away,
and is another process’ responsibility now.  Now, whenever a
process is about to start any work (especially a retry), it should
double-check that the job is registered as being performed by itself.
If not, the process should abort the transaction, make an error
log, and give up on the job.  Write conflict errors on the job should
protect us from the edge cases in this story.The dispatcher’sgetActiveJobsmethod now actually tells you information
about what’s going on in the threads at this instant, rather than what’s
going on in the database.  The poll’sactive jobskeys continues to
report what was truein the databaseas ofthe last poll.  This change
also affects theasync jobsmonitor command.The dispatcher methodgetJobInfo(and the monitor commandasync job)
now returns the name of the queue for the job, the name of the agent for the
job, and whether the job has been, or was reassigned.zc.async events inherit from ‘zc.component.interfaces.IObjectEvent’ instead
of a zc.async specific IObjectEvent (thanks to Satchit Haridas).Added new monitoring and introspection tools: theasyncdbzc.monitor
command (and, for Python, the code in monitordb.py).  This code provides
easy spellings to examine the database’s view of what is happening in
zc.async.  Because it is the database, it also has a much longer historical
view than theasynctools.  The best way to learn about these tools is
to read the extensive documentation provided within zc.monitor by
usingasyncdb helpandasyncdb help <TOOL NAME>.Added new preferred way of filtering agent choices: the newfilterattribute.  Using filters, rather than “choosers,” allows severalasyncdbtools to filter pending jobs based on what an agent is willing to do.  It
also is a smaller contract, and so a filter requires less code than a chooser
in the common case.  On the other hand, using a filter alone doesn’t allow
the agent to try toprefercertain tasks.Deprecated agent.chooseFirst.  It is no longer necesary, since an agent
without a chooser and with a filter of None has the same behavior.  It is
retained for legacy databases.Moved deprecated legacy code to newlegacymodule.Tried to be significantly reduce the chance of spurious timing errors in the
tests, at the expense of causing the tests to take longer to run.monitoring support depends on the new zc.monitor package, which is not Zope
specific.  This means non-Zope 3 apps can take advantage of the monitoring
support.  To use, use the [monitor] target; this only adds simplejson,
zc.ngi, and zc.monitor to the basic dependencies.Make ftesting try to join worker threads, in addition to polling thread,
to try to eliminate intermittent test-runner warnings in ftests that a
thread is left behind.  If the threads do not end, inform the user what jobs
are not letting go.  (thanks to Patrick Strawderman)1.4.2 (2009-07-17)Fix a bug where zc.async.testing._datetime.now did not accept the same
keyword arguments as datetime.datetime, added tests.1.4.1 (2008-07-30)The newserialandparallelhelpers did not allow thepostprocessargument to be a partial closure, and were being naughty.
Fixed.Added tests and demos for advanced features ofserialandparallel.More tweaks to the new Quickstart S5 document.1.4.0 (2008-07-30)Mentioned in ftesting.txt that Zope 3 users should uses zope.app.testing
3.4.2 or newer.  Also added a summary section at the beginning of that file.Added logging of critical messages to __stdout__ forftesting.setUp.
This can help discovering problems in callback transactions.  This uses a new
helper function ,print_logs, in zc.async.testing, which is primarily
intended to be used for quick and dirty debuggingChanged testing.wait_for_result and testing.wait_for_annotation to ignore
ReadConflictErrors, so they can be used more reliably in tests that use
MappingStorage, and other storages without MVCC.Support <type ‘builtin_function_or_method’> for adaptation to Job.Add warning about long commits to tips and tricks.After complaining about a polling dispatcher that is deactivated not really
being dead in the logs, reactivate.No longer use intermediate job to implement the success/failure addCallbacks
behavior.  Introduce an ICallbackProxy that can be used for this kind of
behavior instead.  This change was driven by two desires.Don’t log the intermediate result.  It makes logs harder to read with
unnecessary duplications of pertinent data hidden within unimportant
differences in the log entries.Don’t unnecessarily remember errors in success/failure callbacks.  This can
cause unnecessary failures in unusual situations.The callback proxy accepts callbacks, which are added to the selected job
(success or failure) when the job is selected.This change introduces some hopefully trivial incompatibilities, which
basically come down to the callback being a proxy, not a real job. Use the
convenience propertiessuccessandfailureon the proxy to look at
the respective jobs. After the proxy is evaluated, thejobattribute
will hold the job that was actually run.statusandresultare
conveniences to get the status and result of the selected job.Addparallelandserialconvenience functions to zc.async.job to make
it trivial to schedule and process decomposed jobs.Addstartconvenience function to zc.async.configure to make it trivial
to start up a common-case configuration of a zc.async dispatcher.No longer use protected attributes of callbacks inresumeCallbacks.The “local” code is now moved out from the dispatcher module to
threadlocal.  This is to recognize that the local code is now modified
outside of the dispatcher module, as described in the next bullet.Jobs, when called, are responsible for setting the “local” job value.  This
means that zc.async.local.getJob() always returns the currently running job,
whether it is a top-level job (as before) or a callback (now).Start on S5 QuickStart presentation (see QUICKSTART_1_VIRTUALENV.txt in
package).1.3 (2008-07-04)added “Tips and Tricks” and incorporated into the PyPI page.addedsetUpandtearDownhooks to Job class so that code can run
before and after the main job’s code.  The output ofsetUpis passed as
an argument totearDownso that one can pass state to the other, if
needed.setUpis run immediately before the actual job call.tearDownruns after the transaction is committed, or after it was aborted
if there was a failure.   A retry requested by a retry policy causes the
methods to be run again.  A failure insetUpis considered to be a
failure in the job, as far as the retryPolicy is concerned (i.e., the job
calls the retry policy’sjobErrormethod).  IfsetUpfails, the job
is not called, bittearDownis.tearDownwill fail with a critical
log message, but then processing will continue.using the newsetUpandtearDownhooks, added a Zope 3-specific Job
subclass (see zc.async.z3.Job) that remembers the zope.app.component site and
interaction participants when instantiated. These can be mutated. Then, when
the job is run, thesetUpsets up the site and a security interaction
with the old participants, and then thetearDowntears it all down after
the transaction has committed.changed retry policy logs to “WARNING” level, from “INFO” level.changed many dispatcher errors to “CRITICAL” level from “ERROR” level.added “CRITICAL” level logs for “other” commit retries on the
RetryCommonForever retry policy.addedremovemethod on queue.added helpers for setting up and tearing down Zope 3 functional tests
(ftesting.py), and a discussion of how to write Zope 3 functional tests with
layers (zope.app.testing.functional) in ftesting.txt.remove obsolete retry approach for success/failure callbacks
(completeStartedJobArguments): it is now handled by retry policies.remove odd full-path self-references within the utils module.renamedzc.async.utils.try_transaction_five_timestozc.async.utils.try_five_times.doc improvements and fixes (thanks to Zvezdan Petkovic and Gintautas
Miliauskas).thez3“extra” distutils target now explicitly depends on zope.security,
zope.app.security, and zope.app.component.  This almost certainly does not
increase the practical dependencies of thez3extras, but it does reflect
new direct dependencies of the z3-specific modules in the package.1.2 (2008-06-20)made the log for finding an activated agent report the pertinent queue’s oid
as an unpacked integer, rather than the packed string blob. UseZODB.utils.p64to convert back to an oid that the ZODB will recognize.Bugfix: in failing a job, the job thought it was in its old agent, and thefailcall failed. This is now tested by the first example in new doctestcatastrophes.txt.jobs no longer default to abegin_byvalue of one hour after thebegin_after.  The default now is no limit.Made dispatcher much more robust to transaction errors and ZEO
ClientDisconnected errors.Jobs now use an IRetryPolicy to decide what to do on failure within a job,
within the commit of the result, and if the job is interrupted.  This allows
support of transactional jobs, transactional jobs that critically must be
run to completion, and non-transactional jobs such as communicating with an
external service.The default retry policy supports retries for ClientDisconnected errors,
transaction errors, and interruptions.job.txthas been expanded significantly to show error handling and the
use of retry policies. New filecatastrophes.txtshows handling of other
catastrophes, such as interruptions to polling.job errors now go in the main zc.async.event log rather than in the
zc.async.trace log.  Successes continue to go in the trace log.callback failures go to the main log as a CRITICAL error, by default.handleInterruptis the new protocol on jobs to inform them that they were
active in a dispatcher that is now dead. They either fail or reschedule,
depending on the associated IRetryPolicy for the job. If they reschedule,
this should either be a datetime or timedelta. The job calls the agent’sreschedulemethod. If the timedelta is empty or negative, or the datetime
is earlier than now, the job is put back in the queue with a newputBackmethod on the queue. This is intended to be the opposite ofclaim. Jobs
put in the queue withputBackwill be pulled out before any others.convert to using zope.minmax rather than locally definedAtom.Fix (and simplify) last_ping code so as to reduce unnecessarily writing the
state of the parent DispatcherAgents collection to the database whenever the
atom changed.Depends on new release of zc.twist (1.3)Switched dispatcher’s in-memory storage of job and poll information to be per
job or per poll, respectively, rather than per time period, so as to try and
make memory usage more predictable (for instance, whether a dispatcher is
whipping through lots of jobs quickly, or doing work more slowly).1.1.1 (2008-05-14)more README tweaks.converted all reports from the dispatcher, including the monitor output,
to use “unpacked” integer oids.  This addresses a problem that simplejson
was having in trying to interpret the packed string blobs as unicode, and
then making zc.ngi fall over.  To get the object, then, you’ll need to
useZODB.utils.p64, like this:connection.get(ZODB.utils.p64(INTEGER_OID)), whereINTEGER_OIDindicates the integer oid of the object you want to examine.added several more tests for the monitor code.made theasync jobsmonitor command be “up to the minute”.  Before, it
included all of the new and active jobs from the previous poll; now, it
also filters out those that have since completed.Theasync jobcommand was broken, as revealed by a new monitor test.
Fixed, which also means we need a new version of zope.bforest (1.2) for a new
feature there.1.1 (2008-04-24)Fired events when the IQueues and IQueue objects are installed by the
QueueInstaller (thanks to Fred Drake).Dispatchers make agent threads keep their connections, so each connection’s
object cache use is optimized if the agent regularly requests jobs with
the same objects.README improved (thanks to Benji York and Sebastian Ware).Callbacks are logged at start in the trace log.All job results (including callbacks) are logged, including verbose
tracebacks if the callback generated a failure.Had the ThreadedDispatcherInstaller subscriber stash the thread on the
dispatcher, so you can shut down tests like this:>>> import zc.async.dispatcher
>>> dispatcher = zc.async.dispatcher.get()
>>> dispatcher.reactor.callFromThread(dispatcher.reactor.stop)
>>> dispatcher.thread.join(3)AddedgetQueueto zc.async.local as a convenience (it does what you
could already do:zc.async.local.getJob().queue).Clarified thatIQueue.pullis the approved way of removing scheduled jobs
from a queue in interfaces and README.reports in the logs of a job’s success or failure come before callbacks are
started.Added a section showing how the basic_dispatcher_policy.zcml worked, which
then pushed the former README_3 examples into README_3b.Put ZPL everywhere I was supposed to.Moved a number of helpful testing functions out of footnotes and into
zc.async.testing, both so that zc.async tests don’t have to redefine them
and client packages can reuse them.1.0 (2008-04-09)Initial release.